Liquid Reality Blog from June, 2011

Skip to end of sidebar Go to start of sidebar
How fast is CXF ? - Measuring CXF performance on http, https and jms

From time to time people ask how fast is CXF? Of course this is a difficult question as the measuered speed depends very much on the Hardware of the test setup and on the whole definition of the test.
So I am trying to explain how you can do your own tests and what to do to make sure you get clean results.

What should you keep in mind when doing performance tests with Java?

  • Performance is very much influenced by thread count and request size. So it is a good idea to scale each
  • As long as you have not maxed out at least one resource you can improve the results. Typical resources to check are processor load, memory and network
  • Increase the thread count until you max out a resource. But do not go much higher
  • Always use a warmup phase (~1-2 minutes). Java needs to load classes the first time. On the Sun VM additionally the Hotspot compiler will kick in after some time

Prerequisites

The test project can be found on my github account. You can either download a zip or clone the project with git:
https://github.com/cschneider/performance-tests

As a load generator and measurement tool we use soapUI. Download the free version from the link below:
http://www.soapui.org/

The test plan

We test SOAP/HTTP, SOAP/HTTPS and SOAP/JMS performance using a small but non trivial service. For this case the CustomerService from the wsdl_first example will be used.
Two variables will be changed for the test series. The Soap Message size and the number of sender/listener threads.
The SOAP Message size will be tuned by using a String or variable length. It will be set so the complete SOAP message reaches the desired size.

The payload size can be adjusted by the number of customer records the server sends:

Size payload size
Small 500
Medium 10 KB
Large 1 MB

The second variable is the number of sender and listener Threads. We will test with 5, 10 and 20 Threads. The optimal number of threads
correlates with the number of processor cores. In this case there are two cores. With bigger machines the maximum number of threads should be
higher.

Customerservice SOAP/HTTP performance

For the server side I have prepared a maven project which start the CustomerService implementation from the wsdl_first example on an embedded jetty. We could
also use an external server but in my tests the results were similar and the embedded version can be started very easily.

The number of listener threads can be adjusted in the file src/main/resources/server-applicationContext.xml :

<httpj:threadingParameters minThreads="5" maxThreads="5" />

Start the server:

cd cxf
mvn -Pserver

Start soapUI and load the soapUI project from the file cxf/cxf-performance-soapui-project.xml. The project was built using the wsdl of the CustomerService and contains
test requests and a load test definition. Alternatively a client class is provided that also will give the performance results. But SOAP UI is the more neutral environment.

Now navigate to the Loadtest 1 like shown in screenshot and start the loadtest by clicking on the green arrow. The intersting result is tps (Transactions per seconds). It measures how many Requests/Resonses are processed per second.
At first the number will be quite low but increase steadily. That is because of class loading and optimizations in Java. Let the test run 60 seconds. This was the warmup phase. Now start the test again.

Customerservice SOAP/JMS performance

Testing JMS is much harder than HTTP. SOAP UI supports jms tests but it needs some more configuration than in the http case and did not work so well for me. So
I used the java client for the jms tests.

Additionally there are many tuning possibilities that affect the speed tremendously. For example I was not abler to send more than
700 messages per second in the start as my activemq config was not correctly optimized. When I used the throughput optimized config
the speed was much higher.

Beware though when using the default "activemq-throughput.xml". It limits the size of queues to 1MB and stops the sender when the size is reached.
In my case that meant that my sender was hanging mysteriously. After I set the limit to 100MB my tests worked. See activemq.xml for my configs.

On the ActiveMQ website much more performance tuning tips can be found:http://activemq.apache.org/performance-tuning.html

Environment

It is always important to describe excatly on which configuration the test was run.
All the tests below were run on a Intel Core i5 / 8GB System. Client and Server where on the same machine.

SOAP/HTTP Results

Threads are listener and client threads. CPU load is read from the windows Task Manager. Transactions per Second are the highest number from soapUI.

Threads Size CPU Load Transactions per Second
5 Small 55% 2580
10 Small 100% 3810
20 Small 100% 4072
5 Medium 75% 2360
10 Medium 100% 2840
20 Medium 100% 2820
5 Large 90% 94
10 Large 92% 94
20 Large 95% 84

So it looks like 10 threads is ideal for the test machine with 2 cores and 4 virtual cores. This is quite near the rule of thumb to use double the amount of cores as optimal thread number.
When scaling up the payload size performance drops with the same factor.

SOAP/HTTPS results

Cipher: AES-128 128 Bit key

The payload size can be adjusted by the number of customer records the server sends:

Threads Size CPU Load Transactions per Second
5 Small 60% 2408
10 Small 100% 3310
20 Small 100% 3430
5 Medium 80% 1620
10 Medium 100% 1750
20 Medium 100% 1800
5 Large 100% 34
10 Large 100% 34
20 Large 1000% 34

So it looks like 10 threads is ideal for the test machine with 2 cores and 4 virtual cores. This is quite near the rule of thumb to use double the amount of cores as optimal thread number.
When scaling up the payload size performance drops with the same factor.

SOAP/JMS results

The JMS tests additionally need a broker. I used ActiveMQ 5.5.0 with the activemq.xml that can be found in github repo above.

Using request / reply with a fixed reply queue.

Threads Size CPU Load Transactions per Second
5 Small 100% 1670
10 Small 100% 1650
20 Small 100% 1710
5 Medium 100% 1120
10 Medium 100% 1120
20 Medium 100% 1140
3 Large 75% 30
5 Large 75% 28

Using one way calls

Threads Size CPU Load Transactions per Second (only client) Transactions per Second (client and server)
5 Small 100% 3930 3205
10 Small 100% 3900 3167
20 Small 100% 4200 3166
30 Small 100% 4090 2818

When testing one way calls first only the client was running. So it can be expected that the performance is more than double the performance of
request /response as we do not have to send back a message and there is no server that consumes processor power.

Next the server was also running. This case is as expected about double the performance of request /reply as only half the messages have to be sent / received.

Labels: cxf, jms
Happy Birthday Karaf

Today theApache Karaf project turns one year old. Karaf is a server side OSGi container. It builds upon an OSGi framework (Equinox or Felix) and adds a nice command shell, web console and a lot of management and diagnostic commands to it.

For a project that is so young very much has been achieved. Karaf is used by the integration framework Apache Servicemix as well as other commercial solutions like from Fuse and Talend. It is to date the best server side platform for OSGi I know of and it is still very light weight.

Upcoming features for Version 3.0:

  • OSGi bundle resolver (OBR)
  • Clustering support (Cellar)
  • Enterprise Repository (Links to all the feature files for e.g. ActiveMQ, Camel, CXF) so they can be easily installed

As a side note I am proud to be a committer at Apache Karaf since the start of June.

Labels: karaf