Exposing your service as SOAP Enpoint using Apache Camel

Today I will be telling you about how to expose your service as SOAP endpoint using Apache Camel.


The four steps for it.

1) creating of  cxf-server.xml -- which contain detail about your soap endpoint.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:cxf="http://camel.apache.org/schema/cxf"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:soap="http://cxf.apache.org/bindings/soap"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
                           http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd
                           http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd
                           http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
                           http://cxf.apache.org/bindings/soap http://cxf.apache.org/schemas/configuration/soap.xsd">


    <import resource="classpath:META-INF/cxf/cxf.xml" />
    <import resource="classpath:META-INF/cxf/cxf-servlet.xml" />

   
    <!-- Create SOAP endpoint  -->
    <cxf:cxfEndpoint id="mySoapEndPoint"
                     address="http://localhost:{{port}}/test/SOAP"
                     serviceClass="com.test.Broker">
        <cxf:binding>
            <soap:soapBinding version="1.2"/>  <!-- this allow you accept soap 1.2 request, as  by default camel accept only soap 1.1 request-->
        </cxf:binding>
        <cxf:properties>
            <entry key="dataFormat" value="PAYLOAD"/>
        </cxf:properties>
    </cxf:cxfEndpoint>
   
 
   
</beans>


2) Creation of your camel-server.xml -- defining your route.


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:camel="http://camel.apache.org/schema/spring"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
                           http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring-2.11.0.xsd
                           http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd">

    <import resource="classpath:META-INF/cxf/cxf-config.xml" />


<!-- Define your route here -->
    <camel:camelContext id="camel-server">

        <!-- Externally Exposed Routes -->
        <camel:routeBuilder ref="soapHandlerRoute"/>
     
    </camel:camelContext>

<bean id="soapHandlerRoute"  class="com.test.SoapRoute "/>




3)  Define your route -- the actual implementation.

package com.test;

public class SoapRoute extends RouteBuilder {

@Override
    public void configure() throws Exception {

     from("cxf:bean:mySoapEndPoint")
                .routeId("soapRouteId")
                .convertBodyTo(String.class)
                .end(); //  add your logic here to handle the input soap request.

    }


}


4) Define your dummy endpoint broker class


package com.test;

@WebServiceProvider() @ServiceMode(Mode.PAYLOAD)

public class Broker implements Provider<SOAPMessage>{

    public SOAPMessage invoke(SOAPMessage message) {

        // Requests should not come here as the Camel route will

        // intercept the call before this is invoked.

        throw new UnsupportedOperationException("Placeholder method");

    }

 

}

}


That is it your have exposed your service as SOAP Request and Response.

Refer : cxf-example and cxf.html for more detail

Mongodb and scripting

To do house keeping job on Mongo DB we can use scripting for it

Steps involved will be

1) Write a java script file using the details given in http://docs.mongodb.org/manual/tutorial/write-scripts-for-the-mongo-shell/

2) To run a script use comand : mongo .js

3) If you want to get the shell also when executing like this : mongo testConnction.js --shell

testJsScript.js

` db = connect("localhost:27017/myDatabase"); // db contain the reference of the database printjson(db.getCollectionNames()); // mongodb provided print command print(db.getCollectionNames()); //js print command cursor = db.collection.find(); // this will return a cursor, this cursor can be used to iterate over all the collection object.

you can even use this cursor for updating items, but the catch is you have to fire a query which actually return only the json object which can be directly used in the query.

Mongo DB indexing pitfalls and how to avoid it.

Querying and adding Index to Mongo DB is a bit straight forward. But to use both together to optimize the response time can be a bit tricky.

As adding Index, can help to reduce to response time to find an item, similarly adding unwanted index can lead to other memory and response time-related issue.

So below are some quick  rules/suggestion which should be followed to avoid any pitfalls of indexing

1.  If you have a field X on which you have an index with a unique key constraint.  Then you don't have to have any other index with other fields along with this X field.

e.g

db.myCollection.ensureIndex({X:1},{unique:true})

Then there is no point of having index like:

db.myCollection.ensureIndex({X:1,b:1})  // no need can be removed db.myCollection.ensureIndex({X:1,b:1,c:1}) // no need can be removed

2. If you have a compound index and you want to use this index in your query then make sure your query being fired always have the first element of the index.

e.g

compound index : { w:1,x: 1, y: 1, z: 1 }

Then you can fire any query having the FIRST element (w) to use the index.

wxy wxz wyz xyw yxw zxw

All above query will use the compound index defined above.

But you CAN'T fire any query which does not have the first element of the indexed as it won't use the index.

e.g  

xyz xy zy

For more info read : compound-index   and index-intersection-compound-indexes

3. You should always check your index and write query against it and  not to do vise-verse. If there is no index to cater your query first add the index and then make the query against it.

Camel Technical learnings

Whenever you use a `choice()` in your camel route always use `end()` to end it as opposed to `endChoice()`. Using `endChoice()` drops the exchange from the route.

-------------------------------------

How are bean method is called :
Camel uses a Camel’s method-selection algorithm for selecting a method in the Bean class.
Also uses Bean parameter binding as defined below :
generally this steps are performed:
The BeanProcessor uses the input message to bind its body to the first parameter
of the method of called Bean.

see the different annotation that can be used in defining Bean methods: http://camel.apache.org/parameter-binding-annotations.html

Also see 4.5.1 Binding with multiple parameters in camel in acton.


----------------------------

Configure method of a Routebuilder is called only once at the startup of the camelContext. So any value added to route cant be changed even though  you may have used variable to define them (and variables being dynamically loaded from properties file).

Camel Tuning

1)  Don't use streaming on multicast or splitter


2)  Don't user  allowUseOriginalMessage if you are not going to use the orginal exchange.
scenerios when you should use it when you are going to push back the same message to queue when there is any error.

3) Increase the camel thread pool size if you are using too many splitter or multicast  in your route


<camel:camelContext id="camel-server" allowUseOriginalMessage="false">

        <camel:routeBuilder ref="testRoute"/>

        <camel:threadPoolProfile id="service-threadpool"
                           defaultProfile="true"
                           poolSize="30" maxPoolSize="100" />

    </camel:camelContext>

Learning from NFR (Non Functional Requirement)Testing

Learning from NFR Testing

---------------------------

1) Always use production like topology.
a) In case you cant, make sure people who are going to review it knows about it, suggest them the alternative and if they agree use it else your test will be considered as wrong.
2) All system configuration should be placed in one place and reviewed before testing begin else any change will lead to re-run of test.
3) All data preparation steps should be documented and source script should be attached to the confluence.
4)

Issue you may face and resolution:

Network issue - to check connectivity :

curl -s -v -x '' 'http://127.0.0.1:8080/soap//core/generateOutput?wsdl'

check curl documention for more specific command based on your requirement.

netstat  -o  -v -a | grep tcp    (check other option of netstat for more detail)

Memory and System stats:

 lscpu                          To know the system configuration
ps -aux  | grep <pid >

top         system stats

top -b  | grep <pid>    To see continuous CPU and Mem usage by a process.

pmap -x <pid>          Memory utilization for given process

vmstat                        Monitor virtual memory
free                             Display amount of free and used memory in the system.
pmap                          Display/examine memory map and libraries (so). Usage: pmap pid

sar                              For historical memory usage.

du -H                          For disk usage
df -h            For free disk

Above detail can also be viewed from Monitoring tools if used like Muning, Grafana

For thread analysis of your application :

jmap -dump:format=b,file=<heap-dump-file-path> <pid>

MongoDB stats command :

mongotop

Other things to check :

Access and server log of the application.
Other Application logs.(Jenkins, Mongodb, etc based on the which application is having issue.)


To find the number of thread a process can create : 
 ulimit
ulimit -v 

ulimit -s 

 ps -eLF| grep -c java

To find the number of thread a process created : 
 ps  hu H p  <pid> | wc -l


Force Head / thread Dump
jmap -F -dump:live,format=b,file=<PID>.bin PID

HttpClinet default cookie -- problem

Problem

Problem was we have used load balancing to connect to a third party server, but the sever returns a cookie so clients can have sticky sessions if required.

This results in hitting the same server each time:
curl -s -v -H "Cookie: cookie=R2315085257" 'http://xxxxxxxxx:8080/soap/core/generateOutput?wsdl'

But this round robins:
curl -s -v 'http://xxxxx:8080/soap/core/generateOutput?wsdl'

This means the our application, when accessing third party App through the load balancer, is only using one instance of third party, leaving the other idle.

Diagnosis

Our application  is using HTTPClient and connection pooling. Cookies are not ignored by default.

The cookies sent in the request to Third party Applicaiton via Our application has:

o.a.h.client.protocol.RequestAddCookies - CookieSpec selected: best-match

Hypothesis

Therefore, when a connection is made to Third party Applicaiton via the Loadbalancer  for the first time, the cookie will be set, and then re-used on subsequent connections.

Cookie processing: If an application, such as web spider, does not need to maintain conversational state with the target server, a small performance gain can made by disabling cookie processing. For details on cookie processing please to the HttpClient Cookies Guide.

from: https://hc.apache.org/httpclient-3.x/performance.html#Cookie_processing

Ignore Cookies: This cookie specification ignores all cookies. It should be used to prevent HttpClient from accepting and sending cookies.

from: https://hc.apache.org/httpclient-3.x/cookies.html

Specifying the Specification: There are two ways to specify which cookie specification should be used, either for each HttpMethod instance using the HttpMethodParams, or by setting the default value on CookiePolicy.

from: https://hc.apache.org/httpclient-3.x/cookies.html

Proposed Solution

Therefore, in the RESTClient [1], if we explicitly set the cookie policy to:
HttpMethod method = new GetMethod(); method.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES); we should stop seeing the cooking being sent in subsequent requests from the Our Applicaiton and will load balance to both Third party Applicaiton Servers as required.