Stuck Thread Alert in UNIX - Weblogic

Following is the script which can be used to get email alerts in case of stuck threads in weblogic (installed on UNIX machines)

ScriptName: stuckCheckMS1.sh
#!/bin/ksh
 
Somefile=/export/home/oracle/stuckLogs/ms1_stuckCheck.log
 
count="$(/usr/ucb/ps auxwww | grep ms1 | grep Dweblogic | cut -c 10-15 | tr -d ' ' | xargs /opt/oracle/products/wls10.3.6/jdk1.6.0_45/bin/sparcv9/jstack -F | tee /export/home/oracle/stuckLogs/ms1_stuck.log_`/usr/bin/date +\%m-\%d-\%y-\%H-\%M` | grep STUCK | wc -l)"

if [ $count != 0 ]
then
                        echo "\n Server MS1 has stuck threads on" `/usr/bin/date +\%m-\%d-\%y-\%H-\%M` | mail -s 'STUCK thread on MS1' Some1@gmail.com
> $Sfile
else
                        echo "\n stuck threads status=okay" > $Somefile
fi
 
exit

Admin Password Change - Weblogic

1) Stop the weblogic Managed servers

2) Export the realm

• Login to Admin Console --> Security Realms —> myrealm(Your realm Name)—> Migration(Tab)—> Export (Tab)

3) Stop the Admin Server

4) Rename the data folder   

• mv {DOMAIN-HOME}/servers/AdminServer/data {DOMAIN-HOME}/servers/AdminServer/data-old

5) Take Back-up of DefaultAuthenticatorInit.ldift   

• cd {DOMAIN-HOME}\security
• mv DefaultAuthenticatorInit.ldift DefaultAuthenticatorInit_old.ldift

6)    Set the environment variables (don’t forget "." at the beginning)

• cd {DOMAIN-HOME}/bin
• . /setDomainEnv.sh

7) Reset the password (don’t forget "." at the end) to generate a new DefaultAuthenticatorInit.ldift   

• cd {DOMAIN-HOME}/security
• java weblogic.security.utils.AdminAccount .

8) Update the boot.properties file   

• {DOMAIN-HOME}/servers/AdminServer/security/boot.properties

        username={new-username}
        password={new-password}

9) Start the Admin Server --> startWeblogic.sh

10)    Import realm   

• Login to Admin Console
• Security Realms —> myrealm(Your realm Name) —> Migration(Tab) —> Import (Tab)

11)    Restart Admin Again   

• stopWeblogic.sh
• startWeblogic.sh

12)    Start the Managed Servers    

• Login to Admin console with new password and Start the managed servers

Password Complexity or Password Validation - Weblogic

Step-by-Step approach to implement password complexity/password validation method for weblogic domain:

1) Login to admin console
2) Stop all the managed servers
3) Click Lock & Edit
4) Go to Security Realms and select the name of the realm you are configuring
5) Select Providers > Password Validation
6) The Password Validation Providers table lists the Password Validation providers configured in this security realm
7) Click option “New"
8) Enter any desired name (which will be suitable for your domain/requirement)
9) From the Type drop-down list, select the type of the Password Validation provider and click OK
10) Now the next step is to configure the parameters. As an example, If I select below:
1 number
1 special Character
1 lowercase
1 uppercase
 And total of 8 character password

11) Save and activate changes, followed by admin server restart
12) Create a new user to see whether it meets the requirement or change password for existing user
13) Now Start managed Servers

Upgrade JDK for Weblogic Server

Following is the process to upgrade JDK for Weblogic Servers. I took the example of JDK 1.6 to 1.7:

1) Download and Install 1.7 on the desired machine(s) where your domain is running
2) Shutdown all Servers - Admin, Managed Servers in the domain
3) Shudown Nodemanagers running on the machine
4) Verify that none of the other process are running with exiting JDK 1.6

5) Search for the files in your MW Home directory which are pointing to JDK 1.6 as below:
    find . -type f -name "*.sh" -exec grep -il jdk1.6 {} \;
   
6) The result of search will show the files such as below:

/Middleware/wlserver_10.3/common/bin/commEnv.sh
/Middleware/utils/uninstall/uninstall.sh
/Middleware/utils/quickstart/quickstart.sh
/Middleware/utils/bsu/bsu.sh
/Middleware/domains/my_domain/bin/setDomainEnv.sh

7) Take a backup of all these files

8) Now we need to edit all these files by replacing the location of JDK 1.6 to 1.7

Open each file in vi and use the following to replace all instances at a time:
%s/jdk1.6/jdk1.7/

9) Now start Admin Server and check whether the process is pointing to the new JDK:
/usr/ucb/ps auxwww | grep AdminServer

10) Once Server started, you can verify it through Admin Console as well:
Admin console --> Servers --> Admin Server --> Monitoring --> General --> Java Version

Now start Nodemanager and Managed servers.

11) If there are multiple machines in your domain, make sure you complete the above steps on all the machines before starting servers.

Server & NodeManager Listing

 Following script will display the details on machine Server Name, Process owner, Process ID and NodeManager:

#!/bin/ksh

echo "################################################################################"
echo "# WebLogic Servers"
echo "#  [owner]  [server]  [pid]   "
echo "################################################################################"
ps -ef | grep "[D]weblogic.Name="|while read tmp
do
  owner=`echo $tmp | awk '{ print $1 }'`
  webLogicServer=`echo $tmp | grep -oP "(?<=Dweblogic.Name=)[^ ]+"`
  pid=`echo $tmp | awk '{ print $2 }'`
    echo "$owner $webLogicServer $pid $port"
done|sort|column -t

echo
echo "################################################################################"
echo "# Node Managers"
echo "#       "
echo "################################################################################"
ps -ef | grep "[w]eblogic.NodeManager"|while read tmp
do
  owner=`echo $tmp | awk '{ print $1 }'`
  pid=`echo $tmp | awk '{ print $2 }'`
  port=`echo $tmp | netstat -tlpn 2>/dev/null | grep $pid | awk '{ print $4 }' | tr '\n' ',' | tr ' ' ',' | grep -o ":....," | sort -u | tr -d '\n' | tr -d ':' | sed 's
/,$//'`
  if [ -z "$port" ]; then
    port="null"
  fi
  mw_home=`echo $tmp | grep -oP "(?<=bea.home=)[^ ]+"`
    echo "$owner $mw_home $pid $port"
done|sort|column -t

exit 0

Output will be like this:

./wlsList.sh
################################################################################
# WebLogic Servers
# [owner]  [server]  [pid]
################################################################################
oracle  AdminServer  29385
oracle  MY_server1   1758
oracle  My_server2   2721
oracle  My_server3   4668

################################################################################
# Node Managers
#      
################################################################################
oracle  /opt/admin/fmw  61224  5556

Get Server Status in Weblogic using WLST

First you may create secure key files to avoid using plain text uname and password for weblogic.

Step1:

Run WLST --> /opt/admin/fmw/oracle_common/common/bin/wlst.sh

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline>connect('weblogic','welcome1',url='t3://localhost:7001')
wls:/myDomain/serverConfig> storeUserConfig('/tmp/myUserConfigFile','/tmp/myUserKeyFile')
wls:/myDomain/serverConfig> exit()

Step2:

File name: getServerStatus.py

import sys
import os

sServer=sys.argv[1]

connect(userConfigFile='myUserConfigFile', userKeyFile='myUserKeyFile', url='t3://localhost:7001')
state(sServer)

disconnect()

exit()

Step3:

FileName: getServerState_wls.sh

#!/bin/ksh

usage()
{
 echo "\n!!!!!Usage!!!!!!!"
 echo
 echo "   getServerState_wls.sh "
 echo "\n getServerState_wls.sh myserver1"
 echo
}

if [ $# -ne 1 ]
then
 usage
 exit 2
fi

SERVER=$1

echo "Server= ${SERVER}"
echo

/opt/admin/fmw/oracle_common/common/bin/wlst.sh getServerStatus.py ${SERVER} > /tmp/serverstatus.log
state=`cat /tmp/serverstatus.log | grep -i state`
echo $state


exit 0

Example Run:

 ./getServerState_wls_adp.sh MY_server2
Server= MY_server2


Current state of 'MY_server2' : RUNNING

Managed Server Operations without NM

wls_ms_operations.sh

Usage: ./wls_ms_operations.sh {start|stop|restart} {SERVER-NAME}

#!/bin/bash
server_name=$2
export DOMAIN_LOG=/opt/admin/fmw/

stop() {
        echo "Stopping Managed Server(s)..." $server_name
        echo $SUDO_USER "gave stop command for $server_name at" `/bin/date` > $DOMAIN_LOG/shutdown_$server_name.log
        nohup /opt/admin/fmw/domains/myDomain/bin/stopManagedWebLogic.sh $server_name > $DOMAIN_LOG/shutdown_$server_name.log 2> /dev/null &
        echo "Follow the log @ /opt/admin/fmw/domains/myDomain/servers/${server_name}/logs/"

}

start() {

        echo "Starting Managed Server..." $server_name
        echo $SUDO_USER "gave start command for $server_name at" `/bin/date` > $DOMAIN_LOG/startup_$server_name.log
        nohup /opt/admin/fmw/domains/myDomain/bin/startManagedWebLogic.sh $server_name t3://localhost:7001 2> /dev/null &
        echo "Follow the log @ /opt/admin/fmw/domains/myDomain/servers/${server_name}/logs/"

        exit 1
}

restart() {
        echo "Stopping Managed Server(s)..." $server_name
        echo $SUDO_USER "gave stop command for $server_name at" `/bin/date` > $DOMAIN_LOG/shutdown_$server_name.log
        nohup /opt/admin/fmw/domains/myDomain/bin/stopManagedWebLogic.sh $server_name > $DOMAIN_LOG/shutdown_$server_name.log 2> /dev/null &
        echo "Follow the log @ /opt/admin/fmw/domains/myDomain/servers/${server_name}/logs/"
sleep 10
pid=$(ps auxwww | grep ${server_name}| grep Dweblogic | cut -c 10-15 | tr -d ' ')
if [[ ! -z "$pid" ]]
                then
                        `kill -9 ${pid}`
echo "Killed the process..."
fi
        
echo "Now Starting Managed Server..." $server_name
        echo $SUDO_USER "gave start command for $server_name at" `/bin/date` > $DOMAIN_LOG/startup_$server_name.log
        nohup /opt/admin/fmw/domains/myDomain/bin/startManagedWebLogic.sh $server_name t3://localhost:7001 2> /dev/null &
        echo "Follow the log @ /opt/admin/fmw/domains/myDomain/servers/${server_name}/logs/"
        exit 1

}

case "$1" in

        "start")
                start
                ;;
        "stop")
                stop
                ;;
       "restart")
                restart
                ;;
        *)
                echo $"Usage: $0 {start|stop|restart} {SERVER-NAME}"
                exit 1
                ;;

esac
exit

Admin Server Operations

wls_admin_operations.sh

#!/bin/bash

export SERVER_LOG=/opt/admin/fmw/

start() {
        echo "Starting WebLogic Server(s)..."

echo $SUDO_USER "gave start command for Admin Server at" `/bin/date` > $SERVER_LOG/startup.log

nohup /opt/admin/fmw/domains/myDomain/bin/startWebLogic.sh > $SERVER_LOG/startup.log 2>&1

}

stop() {
        echo "Stopping WebLogic Server(s)..."
echo $SUDO_USER "gave stop command for Admin Server at" `/bin/date` > $SERVER_LOG/shutdown.log
nohup /opt/admin/fmw/domains/myDomain/bin/stopWebLogic.sh > $SERVER_LOG/shutdown.log 2>&1

}

restart() {
        echo "Stopping WebLogic Server(s)..."
        echo $SUDO_USER "gave stop command for Admin Server at" `/bin/date` > $SERVER_LOG/shutdown.log
        nohup /opt/admin/fmw/domains/myDomain/bin/stopWebLogic.sh > $SERVER_LOG/shutdown.log 2>&1
        sleep 40
        echo "Starting WebLogic Server(s)..."
        echo $SUDO_USER "gave start command for Admin Server at" `/bin/date` > $SERVER_LOG/startup.log
        nohup /opt/admin/fmw/domains/myDomain/bin/startWebLogic.sh > $SERVER_LOG/startup.log 2>&1
}

case "$1" in
        "start")
                start
                ;;
        "stop")
                stop
                ;;
        "restart")
                restart
                ;;
        *)
                echo $"Usage: $0 {start|stop|restart}"
                exit 1
                ;;
esac

Clean up Weblogic Server Cache

In many situations we would need to clean up weblogic server cache and we need to be careful while doing this. It's better to have a script which can do our job can help in reducing manual effort/human errors.

Following script can be run using 2 parameters (domain name & server name). This is required because there may be multiple domains in one machine and server name needs to mentioned as we can clean-up any server depending on the need basis.

clearWLCache.sh 

#!/bin/ksh

usage()
{
 echo "\n!!!!!Usage!!!!!!!"
 echo
 echo " clearWLCache.sh "
 echo "\ne.g. clearWLCache.sh myDomain myServer"
 echo
}

if [ $# -ne 2 ]
then
 usage
 exit 2
fi

domain=$1
server=$2
WLS_HOME=/opt/fmw

state=`cut -f1 -d ":" $WLS_HOME/domains/${domain}/servers/${server}/data/nodemanager/${server}.state`
echo "Server state is $state"

if [ $state = RUNNING -o $state = STARTING ]
then
    echo "Server is in $state mode, please shutdown and then use this script "
    exit
else

 if [ -d $WLS_HOME/domains/${domain}/servers/${server} ]
 then
   path=$WLS_HOME/domains/${domain}/servers/${server}
   rm -Rf $path/cache/*
   rm -Rf $path/tmp/*
   echo "Cache clean-up completed successfully"
   exit
 else
   echo ${domain} "or" ${server} "doesn't exists, please check"
   exit
 fi

fi

echo "There was some error, script not executed, please check the command/server name/admin name"
exit

Weblogic Faq's - 2

Q. What is one way SSL?

Ans:  The server is required to present a certificate to the client but the client is not required to present a certificate to the server. To successfully negotiate an SSL connection, the client must authenticate the server, but the server will accept a connection from any client. One-way SSL is common on the Internet where customers want to create secure connections before they share personal data. Often, clients will also use SSL to log on in order for the server can authenticate them.

Q. What is two way SSL?

Ans:  With two-way SSL, the server presents a certificate to the client and the client presents a certificate to the server. Weblogic Server can be configured to require clients to submit valid and trusted certificates before completing the SSL connection.

Setting Up SSL: Main Steps

To set up SSL:

1. Obtain an identity (private key and digital certificates) and trust (certificates of trusted certificate authorities) for Weblogic Server. Use the digital certificates, private keys, and trusted CA certificates provided by the Weblogic Server kit, the CertGen utility, Sun Microsystem's keytool utility, or a reputable vendor such as Entrust or Verisign to perform this step.

2. Store the identity and trust. Private keys and trusted CA certificates which specify identity and trust are stored in a keystore.

3. Configure the identity and trust keystores for Weblogic Server in the Weblogic Server Administration Console.

4. Set SSL configuration options for the private key alias and password in the Weblogic Server Administration Console. Optionally, set configuration options that require the presentation of client certificates (for two-way SSL).

Q. What are the differences among versions of Weblogic 8, 9, 10?

Ans: Few major Differences are listed below:

-          Directory Structure has totally changed compared to 8.1 in 10.3 Refer: http://download.oracle.com/docs/cd/E12840_01/wls/docs103/upgrade/dirstruct.html

-          In WLS 8 configuration information is stored at one place, later versions you’ll find separate XML files

-          In WLS 8 connection pools and data sources are there, WLS 9 & 10 we find connection pools inside data sources

-          In terms of deployment, no need to select the type of application in WLS10 i.e. EJB/Web

-          In WLS 8 execute queues are there, however in WLS 9, 10 work managers are introduced

-          In WLS 9, 10 we have LOCK & EDIT which is not available in earlier versions

-     JMS modules and Sub Deployments are introduced in WLS 9, 10
 

Q. What are the different environments of Weblogic you’ve in your project?

Ans: Please define the exact environment details such as: dev, integration, staging, production

Q. What is the configuration file location of JDBC and config.xml in WLS10.3?

Ans: config.xml file is located in domain_name/config and
JDBC config file: domain_name/config/jdbc

Q. What are default number of threads in 8 and 9 versions of Weblogic.

Ans:  Thread Count 

Development - 15 threads 

Production - 25 threads

Q. How to obtain SSL .pem file.

Ans: SSL .pem (privacy-enhanced mail) files are often required for certificate installations.

First get the der file using keytool command, following the sequence:

keytool -genkey -v -alias test -keyalg RSA -keypass mykeypass -keystore identity.jks -storepass storepass

keytool -selfcert -v -alias test -keypass mykeypass -keystore identity.jks -storetype JKS

keytool -export -v -alias test -file exportedcert.der -keystore identity.jks -storepass storepass

Then convert the der file to pem file.

Q. How to 'Plug-in' for Weblogic on Apache.

Ans: Apache HTTP Server plug-in to Weblogic server allows requests to be proxied. Few high level simple steps are involved as below:

1)      Install Apache HTTP Server Plug-In

2)      Configure the plug-in by editing httpd.conf file which will be located at APACHE_HOME\conf\httpd.conf

3)      Add Weblogic server modules in conf file.

4)      Add an IfModule block with the Weblogic details such as cluster, port details etc…

Memory

1)java.lang.OutOfMemoryError: PermGen space

Information related to the perm space:
The permanent generation is the area of heap that holds all the reflective data of the virtual machine itself, such as class and method objects (also called “method area” in The Java Virtual Machine Specification).
If an application loads “too many” classes then it is possible it will abort with an OutOfMemoryError.

The specific error is: “Exception in thread XXXX java.lang.OutOfMemoryError: PermGen space”.
The JVM itself needs memory to be able to run your application.
The PermGen is information about the classes that make up your application. Web applications and Servlet containers often need more PermGen memory than the JVM defaults to be able to run.
Fixing the java.lang.OutOfMemoryError: PermGen space error: Start your JVM with -XX:MaxPermSize=XXXm where XXX is a number like 256. This will allow the JVM to allocate XXX megabytes of memory for the PermGen space if that space is needed.

2)Information related to determining Memory Requirements:
How do I determine how much memory I need?
Weblogic recommends that you install a minimum of 256 MB of memory for each machine running Weblogic Server that will be handling more than minimal capacity.
If you are expecting a very heavy load then you increase your memory substantially.
A Weblogic Server deployment tracks session objects in memory, either to RAM or swapping to disk. There must be sufficient RAM/disk to store all the session objects. This RAM is accessible to Java through the Java heap.
Out of memory in java heap:
JVM throws java out of memory (java OOM) error if it is not able get more memory in java heap to allocate more java objects. The JVM cannot allocate more java objects if the java heap is full of live objects and it is not able to expand the java heap anymore.

In this situation, the JVM lets the application decide on what to do after throwing the java.lang.OutOfMemoryError.
For example, the application may handle this error and decide to shut down itself in a safe way or decide to run ignoring this error. If the application doesn’t handle this error, then the thread that throws this error will exit (you will not see this thread if you take a java thread dump).

In case of Weblogic server, this error is handled if it is thrown by an execute thread and the error is logged. If this error is being thrown continuously, then the core health monitor thread shuts down the Weblogic server.

Steps to follow in case of java OOM
a) Increase the java heap - Try increasing the java heap to see whether that solves the problem.
b) Caching in the application - If the application caches java objects in memory, then we should make sure that this cache is not growing constantly. There should be a limit for the number of objects in the cache. We can try reducing this limit to see if it reduces the java heap usage.

Java soft references can also be used for data caching as softly reachable objects are guaranteed to be removed when the JVM runs out of java heap.
c) Long living objects - If there are long living objects in the application, and then we can try reducing the life of the objects if possible. For example, tuning HTTP session timeout will help in reclaiming the idle session objects faster.
d) Add ‘-verbosegc’ flag in the java command line - This will print GC activity info to stdout/stderr. Redirect the stdout/stderr to a file. Run the application until the problem gets reproduced.

Make sure that the JVM does the following before throwing java OOM
Full GC run: Does a full GC and all the un-reachable, phantomly, weakly and softly reachable objects are removed and those spaces are reclaimed.
We can check whether full GC was done before the OOM message. A message like this is printed when a full GC is done (format varies depending on the JVM – Check JVM help message to understand the format)
[memory ] 7.160: GC 131072K->130052K (131072K) in 1057.359 ms
The format of the above output is (I will stick to the same format throughout this document) :
[memory ] : GC K->K (K), ms
[memory ] - start time of collection (seconds since jvm start)
[memory ] - memory used by objects before collection (KB)
[memory ] - memory used by objects after collection (KB)
[memory ] - size of heap after collection (KB)
[memory ] - total time of collection (milliseconds)
e) Memory leaks – One example of memory leak is when using database connection pools in application server. When using connection pools, the JDBC statement and resultset objects must be explicitly closed in a finally block.

This is due to the fact that calling close() on the connection objects from pool will simply return the connection back to the pool for re-use and it doesn’t actually close the connection and the associated statement/resultset objects.
It is recommended to follow the coding practices suggested in the following URLs to avoid memory leaks in your application.
http://download-llnw.oracle.com/docs/cd//E13222_01/wls/docs103/jdbc/troubleshooting.html

3)Information related to modifying the classpath:

After installation, Weblogic Server’s classpath is already set, but you may choose to modify it for a number of reasons such as adding a patch to Weblogic Server, updating the version of PointBase you are using, or adding support for Log4j logging.

To apply a patch to ALL of your Weblogic Server domains without the need to modify the classpath of a domain, give the patch JAR file the name, weblogic_sp.jar, and copy it into the WL_HOME/server/lib directory. The commEnv.cmd/sh script will automatically include a JAR named weblogic_sp on the classpath for you.

If you would rather not use the name weblogic_sp.jar for your patch file or you would just like to make sure a JAR file, such as one mentioned below, comes before weblogic.jar on the classpath:

* For ALL domains, edit the commEnv.cmd/sh script in WL_HOME/common/bin and prepend your JAR file to the WEBLOGIC_CLASSPATH environment variable.
* To apply a patch to a SPECIFIC Weblogic Server domain, edit the setDomainEnv.cmd/sh script in that domain’s bin directory, and prepend the JAR file to the PRE_CLASSPATH environment variable.

How WLS Serves Request

1. A client contacts the ListenThread, the entry point into WebLogic Server, which accepts the connection. It then registers the socket with a WLS component known as the SocketMuxer for further processing.

2. The SocketMuxer is responsible for reading and dispatching all client requests to the proper WLS container. It then adds this socket to an internal data structure for processing and makes a request of an ExecuteThreadManager to create a new SocketReaderRequest. This request is then dispatched by the manager to an ExecuteThread

3. As a result, the ExecuteThread becomes a SocketReader thread - it will continually run the SocketMuxer’s processSockets and checks the muxer’s queue to determine if there is work to be done. If an entry exists, it pulls it off the queue and processes it.

4. The SocketReader thread reads the client request and determines the protocol type of this client request, to create a new protocol specific MuxableSocket.

5. The MuxableSocketDiscrminator stores a new MuxableSocket with the implementation matching the protocol of the client. It also returns true to the SocketReader to notify it that the message is complete and it can be dispatched.

6. MuxableSocketDiscriminator re-registers the protocol specific version of the MuxableSocket that was created earlier. The net result is that “Step 2” is repeated, and a new protocol specific MuxableSocket is placed in the SocketMuxer’s queue for processing.

7. A socket reader will get the new protocol specific MuxableSocket off the queue and read it. It will then checks to see if the message is complete, based on the protocol. If it is, it will invoke the protocol specific MuxableSocketDiscriminator

8. Before the work requested by the client can be performed, there may be many iterations of “step 7”. This is determined by the protocol – for example, t3 will read a portion of the message, dispatch it so it can act upon the portion of the protocol read thus far.

9. The subsystem will create an ExecuteRequest and send it to an ExecuteThreadManager for processing. The request is dispatched to an ExecuteThread, and the result is returned to the client.


From a high level overview’s perspective, the SocketMuxer can be explained as follows. Each and every socket connection that comes into WebLogic Server is “registered” with the SocketMuxer - which then maintains a list of these connections, each represented by a form of the MuxableSocket interface. It then becomes the responsibility of the SocketMuxer to read and dispatch each client request to the appropriate subsystem. This is a fairly elaborate process, which is illustrated by steps 2 through 8 above.

There are only a few key things to know about the SocketMuxer:

First, it has a data structure in which it stores a socket entry for each client connected to WebLogic Server.
Second, a “socket reader” is the main component of the SocketMuxer - which really is just an execute thread that is running the SocketMuxer’s processSockets() method.

Third, the SocketMuxer does most of its work through the only interface it knows how to operate on – the MuxableSocket interface.

Socket Reader:

A SocketReaderRequest is merely an implementation of an ExecuteRequest, which is sent to the ExecuteThreadManger by the invocation of the registerSocket(). When the ExecuteThread invokes the execute() method of the SocketReaderRequest, the SocketMuxer’s processSockets() method is invoked.
So, a socket reader thread is simply a normal execute thread which runs the main processing method of the SocketMuxer, processSockets().


The acceptBacklog parameter of Weblogic server is passed to ServerSocket. The value of acceptBacklog means "The maximum queue length for incoming connection indications (a request to connect) is set to the backlog parameter. If a connection indication arrives when the queue is full, the connection is refused. "

Thus if too many connects come on the server at the same time, the server would queue this connects and process them one at a time. The value does not mean that only that many clients can connect to the server.

It does not limit the number of connections made. It limits the number of potential connections that can lie in the backlog queue. So for e.g.: AcceptBacklog is 2. If hundreds of connections were made to the server and the server has one thread to accept new connections.

This thread accepts the new connection and dispatches it to a new thread and then goes back to listening to new connections. Sample code is

while (true) {

Socket sock = serversocket.accept(); // Line 1
new MyThread(sock).run(); // Line 2

}

Here the thread accepts a new connection at line1. Dispatches to new thread in line 2. Evaluates the while expression and goes back to line 1. So in between the time it takes for it to get back to line 1(say T1) many new connections requests are made by the clients. These new connections lie on the accept backlog queue and this queue length is controlled by the accept backlog parameter.

If the queue length is 2, and between this time T1 several hundred connections are made to the server only 2 would get accepted and rest of them rejected. For rejection there must be too many simultaneous requests to the server, if it’s not simultaneous then the chances of queue getting full is less.

Note: Thanks to Sreedevi for helping me to provide this valuable information.

Tweaking JVM parameters for admin and managed server

If we want to give different memory arguments (preferably -Xms and -Xmx) for Admin server and Managed server, both are on same box.

We cannot provide the arguments in startManagedWeblogic.cmd script since it calls startWeblogic file which in turn calls setDomainEnv script.

If we set the parameters in setDomainEnv script then it will get reflected on both the servers (Admin and Managed server).

Solution: To tweak the JVM parameters, create the scripts as follows, for example:

startAdminWithCustomMemArgs.cmd that does the following:

set USER_MEM_ARGS=-Xms256m -Xmx512m
startWeblogic.cmd

And similarly for a managed instance:

startMgdWithCustomMemArgs.cmd:

set USER_MEM_ARGS=-Xms512m -Xmx1024m
startManagedWeblogic.cmd MS1

In the second script above “MS1” is the name of the managed server.

Weblogic FAQs

-->I wanna share some common Questions related to Weblogic Server which would be useful to any weblogic professional.

Cluster Related Questions: Q. What is weblogic cluster?Ans: A Weblogic cluster is logical group of weblogic server instances working together for coordination. A cluster appears to clients to be a single WebLogic Server instance whether it is a web client or a java application.

Q. What is load balancing?
Ans: Equally distributing the load across the servers to serve the application in a smoother way.
Algorithms used for load balancing are: Round Robin, Weight based, and Random.
Server Affinity load balancing algorithms (round-robin-affinity, weight-based-affinity, and random-affinity)

Q. How does a server know when another server is unavailable?
Ans: WebLogic Server uses two mechanisms to determine if a given server instance is unavailable. Each WebLogic Server instance in a cluster uses multicast to broadcast regular “heartbeat” messages that advertise its availability. By monitoring heartbeat messages, server instances in a cluster determine when a server instance has failed. The other server instances will drop a server instance from the cluster, if they do not receive three consecutive heartbeats from that server instance
WebLogic Server also monitors socket errors to determine the availability of a server instance.
For example, if server instance A has an open socket to server instance B, and the socket unexpectedly closes, server A assumes that server B is offline.

Q. How do clients learn about new WebLogic Server instances?
Ans: Once a client has done a JNDI lookup and begins using an object reference, it finds out about new server instances only after the cluster-aware stub has updated its list of available servers.

Q. How do stubs work in a WebLogic Server cluster?
Ans: Clients that connect to a WebLogic Server cluster and look up a clustered object obtain a replica-aware stub for the object. This stub contains the list of available server instances that host implementations of the object. The stub also contains the load balancing logic for distributing the load among its host servers.

Q. What is multicast test?
Ans: The Multicast Test is a network paradigm in which machines communicate each other. This utility helps you debug multicast problems when configuring a Weblogic Cluster. The utility sends out multicast packets and returns information about how effectively multicast is working on your network.
As per the general recommendation do not perform the Multicast test when Managed servers are running.

This is because Managed servers communicate each other in Cluster to share the actual data and when we do Multicast test there will be confusion between actual data and test data.

If your environment is in Production and you cannot bring Managed servers down then you can change Multicast test address and can perform the test to see the results.

You can also run some networking diagnostics to make sure there are no hardware issues affecting communication among the cluster nodes.

At a minimum, the ping utility can be used to verify that network communication is unhindered (in both directions) between each of the server addresses.

Q. Which services cannot be clustered?
Ans: File services, Time services. They cannot be deployed homogenously through a cluster and they do not benefit from any of the clustering services.

Q. Which protocol is used for multicast?
Ans: UDP - User Datagram Protocol

Q. What is the range of multicast address assignments?
Ans: The multicast addresses are in the range 224.0.0.0 through 239.255.255.255.

Q. What is difference between unicast and multicast?
Ans: Weblogic server provides an alternative to using multicast to handle cluster messaging and communications. Unicast configuration is much easier because it does not require cross network configuration that multicast requires.
Additionally, it reduces potential network errors that can occur from multicast address conflicts. Multicast is communication between a single sender and multiple receivers on a network. Typical uses include the updating of mobile personnel from a home office and the periodic issuance of online newsletters.
Multicast enables a single device to communicate with a specific set of hosts, not defined by any standard IP address and mask combination. This allows for communication that resembles a conference call. Anyone from anywhere can join the conference, and everyone at the conference hears what the speaker has to say. The speaker's message isn't broadcasted everywhere, but only to those in the conference call itself. A special set of addresses is used for multicast communication.
Unicast is communication between a single sender and a single receiver over a network. Unicast servers provide a stream to a single user at a time, while multicast servers can support a larger audience by serving content simultaneously to multiple users.
Multicasting exists as UDP multicasting, whereas non-multicast UDP (and TCP) messages are called unicast. The next thing to know is that multicast will often not be sent over a router to another network.
The TTL is low for most multicast packets. TTL on IP packets refers to the maximum number of network hops that a packet can make to get to its destination. Unicast packets typically are allowed to cross about 30 networks.
The above mentions the basic differences between multicast and unicast communication. The unicast communication should be faster as it implements the point to point communication within servers. Comparatively both the multicast and unicast communications have their own advantages as well as disadvantages.

Thread Related Questions:

Q. How are new threads assigned?
Ans: As work enters a Weblogic Server, it is placed in an execute queue. This work is then assigned to a thread within the queue that performs the work.
By default, a new server instance is configured with a default execute queue, weblogic.kernel.default, that contains 15 threads (in development) and 25 threads (in production).
In addition, Weblogic Server provides two other pre-configured queues:
weblogic.admin.HTTP—Available only on Administration Servers, this queue is reserved for communicating with the Administration Console; you cannot reconfigure it.
weblogic.admin.RMI—Both Administration Servers and Managed Servers have this queue; it is reserved for administrative traffic; you cannot reconfigure it.
Unless you configure additional execute queues, and assign applications to them, web applications and RMI objects use weblogic.kernel.default.

Q. Is there an ability to configure the container to kill stuck threads after X amount of time? Ans: Currently, Weblogic reports that a thread is stuck after X amount of time, but the thread keeps running.
Once the thread is stuck or processing it cannot be killed.
If they are not stuck or processing the request then they are called idle thread that is waiting for request (waitforrequest() method: A thread invokes this method when it is waiting for work to be assigned to it by its manager).

Q. Is there a way to manually kill a stuck thread?
Ans: No, once thread is created it would be reused but cannot be killed.

Q. Why does the user “WLS Kernel” own so many threads?
Ans: WLS Kernel owns only the number of the threads which are configured as a default.

Q. What is “WLS Kernel” doing with these threads?
Ans: As specified earlier it is kept in default execute queue and then assigned to a thread within the queue that performs the work.

Q. Can we make these threads available to the application?
Ans: No we cannot do that.


Miscellaneous Questions:
Q. App server, Web server: What's the difference?
Ans:
The Web server
A Web server handles the HTTP protocol. When the Web server receives an HTTP request, it responds with an HTTP response, such as sending back an HTML page. To process a request, a Web server may respond with a static HTML page or image, send a redirect, or delegate the dynamic response generation to some other program such as CGI scripts, JSPs (JavaServer Pages), servlets, ASPs (Active Server Pages), server-side JavaScripts, or some other server-side technology. Whatever their purpose, such server-side programs generate a response, most often in HTML, for viewing in a Web browser.
When a request comes into the Web server, the Web server simply passes the request to the program best able to handle it. The Web server doesn't provide any functionality beyond simply providing an environment in which the server-side program can execute and pass back the generated responses. The server-side program usually provides for itself such functions as transaction processing, database connectivity, and messaging.
While a Web server may not itself support transactions or database connection pooling, it may employ various strategies for fault tolerance and scalability such as load balancing, caching, and clustering — features oftentimes erroneously assigned as features reserved only for application servers.
The application server
An application server exposes business logic to client applications through various protocols, possibly including HTTP. While a Web server mainly deals with sending HTML for display in a Web browser, an application server provides access to business logic for use by client application programs. The application program can use this logic just as it would call a method on an object (or a function in the procedural world).
Such application server clients can include GUIs (graphical user interface) running on a PC, a Web server, or even other application servers. The information traveling back and forth between an application server and its client is not restricted to simple display markup. Instead, the information is program logic. Since the logic takes the form of data and method calls and not static HTML, the client can employ the exposed business logic however it wants.
In most cases, the server exposes this business logic through a component API, such as the EJB (Enterprise JavaBean) component model found on J2EE (Java 2 Platform, Enterprise Edition) application servers. Moreover, the application server manages its own resources. Such gate-keeping duties include security, transaction processing, resource pooling, and messaging. Like a Web server, an application server may also employ various scalability and fault-tolerance techniques.

Q. What are tlog files?
Tlog files are the files used by weblogic to keep a trace of current transactions.
For example, when an instance is restarted, the tlog files are used to perform the second step of a two-phase commit on a transaction that was in progress. Sometimes, when a transaction is corrupted, you've got to delete these files to get rid of the phantom transaction. There is no need to restart a server to have them recreated.

Q. Why do I get “NoClassDefFound”/“Too Many Open files” messages on Solaris?
Problem: When I am using WebLogic Server on Solaris and try to run my application, I get a “NoClassDefFound” error, although the class causing the error does exist and is in the right directory. In fact, there are other classes in the same directory that are getting loaded. I also get a “Too many open files” error.
Ans: On Solaris, each user account has a certain limited number of file descriptors. You can find out how many file descriptors you have with the limit command in csh.
You can increase file descriptors if you have enough privileges with the ulimit command in the csh. Otherwise, ask your system administrator to increase the file descriptors available to your processes.

Q. How can I speed up connection responses?
Ans: Connection delays are often caused by DNS problems. WebLogic performs a reverse lookup on the hostname from which a new connection is made. If the DNS reverse lookup is not working properly because the connection is coming from a proxy server, it could be responsible for the delay. You might want to work with your system administrator to determine whether DNS and the third-party networking software are working properly. Try writing a simple server program that performs a reverse lookup on any connection made to it. If that lookup is delayed, then you know that the proxy server is the source of the problem.

Q. Differences between Sun JVM and JRockit
Ans: Users cannot explicitly set the PermSize: The memory JRockit uses can conceptually be divided into only two categories: heap and native memory. Unlike Sun JVM,
JRockit has no special memory areas (like MaxPermSize for “permanent generation” allocations). Java objects are created on the heap, and all other memory allocations are done in native memory (which is considered non-heap memory).
Heap + Native memory is approximately = JVM process size.
JRockit uses more native memory than Sun’s JVM, meaning it usually will not support equivalent heap sizes.
JRockit doesn’t have:
- java code interpretation mode, only compilation mode
(Java code compiled into byte code itself compiled into platform specific machine code)
- the notion a permanent generation set via MaxPermSize. This part of the memory is unbounded in the native memory (outside the java heap) and grows as needed
JRockit has:
- More diagnostic tools than other JVMs such as JRA
- Management console with no overhead on performance
- Better performance on Intel architectures than other VMs
- Higher memory usage for better performance
Advantages of JRockit:
What gives it the performance advantage?
Sun’s HotSpot JVM interprets all byte-code and compiles the hotspots (frequently called methods).
- This is its default mode of execution (-Xmixed)
- (–Xint) has to be specified to disable the HotSpot compiler and run in “interpreted-only” mode.
JRockit compiles all byte-code and optimizes the hotspots.
- The compilation gives JRockit its performance advantage and cannot be disabled. All methods that JRockit encounters for the first time in a run are converted into an internal representation of code (“code generation”) which is then compiled.
- The compilation is also what causes JRockit to have a larger footprint, since it uses native memory for code generation, generated code, optimization, garbage collection, thread allocation, etc. JSPs are converted into Java code, which is then compiled. JSPs are generally complex and produce large Java code, so they tend to be especially high consumers of native memory. An application with lots of JSPs can cause JRockit to exhaust the native memory and swamp the virtual process space.
- The optimization can be turned off by specifying the command-line option “-Xnoopt”. Historically, optimization has not entirely been reliable and can cause crashes and performance problems, so, if the customer is willing, –Xnoopt is a standard workaround that is worth trying blindly.

Q. What is the function of T3 in WebLogic Server?
Ans: T3 provides a framework for WebLogic Server messages that support for enhancements.
These enhancements include abbreviations and features, such as object replacement, that work in the context of WebLogic Server clusters and HTTP and other product tunneling.
T3 predates Java Object Serialization and RMI, while closely tracking and leveraging these specifications. T3 is a superset of Java Object. Serialization or RMI; anything you can do in Java Object Serialization and RMI can be done over T3.
T3 is mandated between WebLogic Servers and between programmatic clients and a WebLogic Server cluster. HTTP and IIOP are optional protocols that can be used to communicate between other processes and WebLogic Server. It depends on what you want to do. For example, when you want to communicate between
- A browser and WebLogic Server-use HTTP
- An ORB and WebLogic Server-IIOP.


Q. Can I refresh static components of a deployed application without having to redeploy the entire application?
A. Yes. You can use weblogic.Deployer to specify a component and target a server, using the following syntax:
java weblogic.Deployer -adminurl http://admin:7001 -name appname -targets
server1,server2 -deploy jsps/*.jsp
Q. How do XA and non-XA drivers differ in distributed transactions?
Ans: The differences between XA and non-XA JDBC drivers are:
Atomicity Guarantee. An XA driver implements the XAResource interface and can participate fully in the 2PC protocol driven by the WLS Transaction Manager. This guarantees atomicity of updates across multiple participating resources.
However, a non-XA driver does not implement the XAResource interface and cannot fully participate in the 2PC protocol. When using a non-XA driver in a distributed transaction, WLS implements the XAResource wrapper on behalf of the non-XA driver. If the data source property enableTwoPhaseCommit is set to true, then the WLS XAResource wrapper returns XA_OK when the Transaction Manager invokes the prepare() method. When the Transaction Manager invokes commit() or rollback() during the second phase, the WLS XAResource wrapper delegates the commit() or rollback() call to the non-XA JDBC connection. Any failure during commit() or rollback() results in heuristic exceptions. Application data may be left in an inconsistent state as a result of heuristic failure.
Redirecting Connections. A non-XA driver can be configured to perform updates in the same distributed transaction from more than one process. WLS internally redirects the JDBC calls made from different processes to the same physical JDBC connection in one process. However, when you use a XA driver, no such redirection will be done. Each process will use its own local XA database connection, and the database ensures that all the distributed updates made in the same distributed transaction from different processes will be committed atomically.
Connection Management. Whether you are using the non-XA driver or XA driver in distributed transactions, WLS implements JDBC wrappers that intercept all the JDBC calls and obtains a physical JDBC connection from the connection pool on demand.
When you use a non-XA driver in distributed transactions, in order to ensure that updates made from different processes are committed atomically, WLS associates the same physical JDBC connection with the distributed transaction until it is committed or rolled back. As a result, the number of active distributed transactions using the non-XA connection pool is limited by the maximum capacity of the JDBC connection pool.
When you use an XA driver, the connection management is more scalable. WLS does not hold on to the same physical XA connection until the transaction is committed or rolled back. In fact, in most cases, the XA connection as only held for the duration of a method invocation. WLS JDBC wrappers intercept all JDBC calls and enlist the XAResource associated with the XA connection on demand. When the method invocation returns to the caller, or when it makes another call to another server, WLS delists the XAResource associated with the XA connection.
WLS also returns the XA connection to the connection pool on delistment if there are no open result sets. Also, during commit processing, any XAResource object can be used to commit any number of distributed transactions in parallel. As a result, neither the number of active distributed transactions using the XA connection pool nor the number of concurrent commit/rollbacks is limited by the maximum capacity of the connection pool. Only the number of concurrent database access connections is limited by the maximum capacity of the connection pool.

Q. Can I use more than one non-XA connection pool in distributed transactions?
A. No. Even if you set EnableTwoPhaseCommit=true for both TxDataSources of the connection pools, attempting to use two non-XA connection pools in the same distributed transaction will result in:
"java.sql.SQLException: Connection has already been created in this tx context for pool named. Illegal attempt to create connection from another pool: “
when you attempt to get the connection from the second non-XA connection pool.

Q. How does the Connection Reserve Timeout parameter on a datasource work?
A: Weblogic uses this parameter to wait for a connection to become available in the pool. The wait starts when your application requests a connection from a connection pool backing the datasource. If the pool has no connection free, it will wait up to the set timeout to see if a connection returned to the pool by some other thread. If this happens, you will get that one. If no connection has become available within the timeout period you will get an exception.

A better approach is increasing size of the connection pool so that it can accommodate all requests for connections. This parameter has no effect if there are timeouts while testing reserved connections.

 
Q. What is node manager?
Ans: Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart Administration Server and Managed Server instances from a remote location.

Q. What are the modes available in WLST?
Ans: Interactive Mode, Script Mode, and Embedded Mode
Interactively, on the command line— Interactive Mode
In batches, supplied in a file— Script Mode
Embedded in Java code— Embedded Mode

Q. What are the files which get generated when we click on “lock & edit” option of weblogic console?
Ans: 2 files - edit.lok & pending

Q. Where will be user credentials stored?
Ans: LDAP

Q. What are the causes of "java.net.SocketException: Broken pipe"

Ans: These kinds of errors are most common in Web Applications.    

Scenario 1:

One very common scenario that would cause broken pipe errors is that, you have a browser client submitting a request to the server, while the server is writing the response back to the browser client; the user hits the Stop button or simply kills the browser.

Since the server is still writing the response back, the "pipe" to the client is gone, resulting in broken pipe. This in itself is not an error, but from the specifications mandates the server to regard this as an error, hence WebLogic has to report this and log this to the server log.

This is quite common, since at times, the user's bandwidth might become low, or server takes a bit of time in processing the request, or simply that the result to be written back to the response from the server is large. Lot of times, users does not want to wait, and click the Stop button in browser.

This error indicates a communication problem between the client and the server. The socket from the server back to the client is dead.

This can occur from events such as:

a) The browser (client) breaks the socket connection by hitting the stop button, or the user issuing another HTTP request prior to the first request finishing.

b) Network congestion, latency, etc.

c) Firewalls, timeouts, etc.    

The error could happen if the number of files that can be open has been reached.

So, these errors can be indicative of user generated errors beyond your control, network -- internet problems beyond your control, or your own network configuration that is causing problems.

For this issue you can also test your production network topology and configuration vs. your test configuration, as the later is usually less complex and restrictive.

Scenario 2: A broken pipe also occurs because your JDBC connection has expired for some reason, either the DB has been restarted or the connection has timed out. Sometimes, improper driver will also cause this problem.