Blog
Skip to end of metadata
Go to start of metadata

From time to time people ask how fast is CXF? Of course this is a difficult question as the measuered speed depends very much on the Hardware of the test setup and on the whole definition of the test.
So I am trying to explain how you can do your own tests and what to do to make sure you get clean results.

What should you keep in mind when doing performance tests with Java?

  • Performance is very much influenced by thread count and request size. So it is a good idea to scale each
  • As long as you have not maxed out at least one resource you can improve the results. Typical resources to check are processor load, memory and network
  • Increase the thread count until you max out a resource. But do not go much higher
  • Always use a warmup phase (~1-2 minutes). Java needs to load classes the first time. On the Sun VM additionally the Hotspot compiler will kick in after some time

Prerequisites

The test project can be found on my github account. You can either download a zip or clone the project with git:
https://github.com/cschneider/performance-tests

As a load generator and measurement tool we use soapUI. Download the free version from the link below:
http://www.soapui.org/

The test plan

We test SOAP/HTTP, SOAP/HTTPS and SOAP/JMS performance using a small but non trivial service. For this case the CustomerService from the wsdl_first example will be used.
Two variables will be changed for the test series. The Soap Message size and the number of sender/listener threads.
The SOAP Message size will be tuned by using a String or variable length. It will be set so the complete SOAP message reaches the desired size.

The payload size can be adjusted by the number of customer records the server sends:

Size

payload size

Small

500

Medium

10 KB

Large

1 MB

The second variable is the number of sender and listener Threads. We will test with 5, 10 and 20 Threads. The optimal number of threads
correlates with the number of processor cores. In this case there are two cores. With bigger machines the maximum number of threads should be
higher.

Customerservice SOAP/HTTP performance

For the server side I have prepared a maven project which start the CustomerService implementation from the wsdl_first example on an embedded jetty. We could
also use an external server but in my tests the results were similar and the embedded version can be started very easily.

The number of listener threads can be adjusted in the file src/main/resources/server-applicationContext.xml :

Start the server:

cd cxf
mvn -Pserver

Start soapUI and load the soapUI project from the file cxf/cxf-performance-soapui-project.xml. The project was built using the wsdl of the CustomerService and contains
test requests and a load test definition. Alternatively a client class is provided that also will give the performance results. But SOAP UI is the more neutral environment.

Now navigate to the Loadtest 1 like shown in screenshot and start the loadtest by clicking on the green arrow. The intersting result is tps (Transactions per seconds). It measures how many Requests/Resonses are processed per second.
At first the number will be quite low but increase steadily. That is because of class loading and optimizations in Java. Let the test run 60 seconds. This was the warmup phase. Now start the test again.

Customerservice SOAP/JMS performance

Testing JMS is much harder than HTTP. SOAP UI supports jms tests but it needs some more configuration than in the http case and did not work so well for me. So
I used the java client for the jms tests.

Additionally there are many tuning possibilities that affect the speed tremendously. For example I was not abler to send more than
700 messages per second in the start as my activemq config was not correctly optimized. When I used the throughput optimized config
the speed was much higher.

Beware though when using the default "activemq-throughput.xml". It limits the size of queues to 1MB and stops the sender when the size is reached.
In my case that meant that my sender was hanging mysteriously. After I set the limit to 100MB my tests worked. See activemq.xml for my configs.

On the ActiveMQ website much more performance tuning tips can be found:http://activemq.apache.org/performance-tuning.html

Environment

It is always important to describe excatly on which configuration the test was run.
All the tests below were run on a Intel Core i5 / 8GB System. Client and Server where on the same machine.

SOAP/HTTP Results

Threads are listener and client threads. CPU load is read from the windows Task Manager. Transactions per Second are the highest number from soapUI.

Threads

Size

CPU Load

Transactions per Second

5

Small

55%

2580

10

Small

100%

3810

20

Small

100%

4072

5

Medium

75%

2360

10

Medium

100%

2840

20

Medium

100%

2820

5

Large

90%

94

10

Large

92%

94

20

Large

95%

84

So it looks like 10 threads is ideal for the test machine with 2 cores and 4 virtual cores. This is quite near the rule of thumb to use double the amount of cores as optimal thread number.
When scaling up the payload size performance drops with the same factor.

SOAP/HTTPS results

Cipher: AES-128 128 Bit key

The payload size can be adjusted by the number of customer records the server sends:

Threads

Size

CPU Load

Transactions per Second

5

Small

60%

2408

10

Small

100%

3310

20

Small

100%

3430

5

Medium

80%

1620

10

Medium

100%

1750

20

Medium

100%

1800

5

Large

100%

34

10

Large

100%

34

20

Large

1000%

34

So it looks like 10 threads is ideal for the test machine with 2 cores and 4 virtual cores. This is quite near the rule of thumb to use double the amount of cores as optimal thread number.
When scaling up the payload size performance drops with the same factor.

SOAP/JMS results

The JMS tests additionally need a broker. I used ActiveMQ 5.5.0 with the activemq.xml that can be found in github repo above.

Using request / reply with a fixed reply queue.

Threads

Size

CPU Load

Transactions per Second

5

Small

100%

1670

10

Small

100%

1650

20

Small

100%

1710

5

Medium

100%

1120

10

Medium

100%

1120

20

Medium

100%

1140

3

Large

75%

30

5

Large

75%

28

Using one way calls

Threads

Size

CPU Load

Transactions per Second (only client)

Transactions per Second (client and server)

5

Small

100%

3930

3205

10

Small

100%

3900

3167

20

Small

100%

4200

3166

30

Small

100%

4090

2818

When testing one way calls first only the client was running. So it can be expected that the performance is more than double the performance of
request /response as we do not have to send back a message and there is no server that consumes processor power.

Next the server was also running. This case is as expected about double the performance of request /reply as only half the messages have to be sent / received.