Learning from NFR (Non Functional Requirement)Testing

Learning from NFR Testing

---------------------------

1) Always use production like topology.
a) In case you cant, make sure people who are going to review it knows about it, suggest them the alternative and if they agree use it else your test will be considered as wrong.
2) All system configuration should be placed in one place and reviewed before testing begin else any change will lead to re-run of test.
3) All data preparation steps should be documented and source script should be attached to the confluence.
4)

Issue you may face and resolution:

Network issue - to check connectivity :

curl -s -v -x '' 'http://127.0.0.1:8080/soap//core/generateOutput?wsdl'

check curl documention for more specific command based on your requirement.

netstat  -o  -v -a | grep tcp    (check other option of netstat for more detail)

Memory and System stats:

 lscpu                          To know the system configuration
ps -aux  | grep <pid >

top         system stats

top -b  | grep <pid>    To see continuous CPU and Mem usage by a process.

pmap -x <pid>          Memory utilization for given process

vmstat                        Monitor virtual memory
free                             Display amount of free and used memory in the system.
pmap                          Display/examine memory map and libraries (so). Usage: pmap pid

sar                              For historical memory usage.

du -H                          For disk usage
df -h            For free disk

Above detail can also be viewed from Monitoring tools if used like Muning, Grafana

For thread analysis of your application :

jmap -dump:format=b,file=<heap-dump-file-path> <pid>

MongoDB stats command :

mongotop

Other things to check :

Access and server log of the application.
Other Application logs.(Jenkins, Mongodb, etc based on the which application is having issue.)


To find the number of thread a process can create : 
 ulimit
ulimit -v 

ulimit -s 

 ps -eLF| grep -c java

To find the number of thread a process created : 
 ps  hu H p  <pid> | wc -l


Force Head / thread Dump
jmap -F -dump:live,format=b,file=<PID>.bin PID

HttpClinet default cookie -- problem

Problem

Problem was we have used load balancing to connect to a third party server, but the sever returns a cookie so clients can have sticky sessions if required.

This results in hitting the same server each time:
curl -s -v -H "Cookie: cookie=R2315085257" 'http://xxxxxxxxx:8080/soap/core/generateOutput?wsdl'

But this round robins:
curl -s -v 'http://xxxxx:8080/soap/core/generateOutput?wsdl'

This means the our application, when accessing third party App through the load balancer, is only using one instance of third party, leaving the other idle.

Diagnosis

Our application  is using HTTPClient and connection pooling. Cookies are not ignored by default.

The cookies sent in the request to Third party Applicaiton via Our application has:

o.a.h.client.protocol.RequestAddCookies - CookieSpec selected: best-match

Hypothesis

Therefore, when a connection is made to Third party Applicaiton via the Loadbalancer  for the first time, the cookie will be set, and then re-used on subsequent connections.

Cookie processing: If an application, such as web spider, does not need to maintain conversational state with the target server, a small performance gain can made by disabling cookie processing. For details on cookie processing please to the HttpClient Cookies Guide.

from: https://hc.apache.org/httpclient-3.x/performance.html#Cookie_processing

Ignore Cookies: This cookie specification ignores all cookies. It should be used to prevent HttpClient from accepting and sending cookies.

from: https://hc.apache.org/httpclient-3.x/cookies.html

Specifying the Specification: There are two ways to specify which cookie specification should be used, either for each HttpMethod instance using the HttpMethodParams, or by setting the default value on CookiePolicy.

from: https://hc.apache.org/httpclient-3.x/cookies.html

Proposed Solution

Therefore, in the RESTClient [1], if we explicitly set the cookie policy to:
HttpMethod method = new GetMethod(); method.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES); we should stop seeing the cooking being sent in subsequent requests from the Our Applicaiton and will load balance to both Third party Applicaiton Servers as required.