Liquid Reality Blog from May, 2011

Skip to end of sidebar Go to start of sidebar
Monitoring your CXF Services Calls with Talend ESB Standard Edition

The Service Activity Monitoring (SAM) component logs service calls made with the Apache CXF Framework to a database. Typical use cases are tracking of errors, creating statistics and providing a basis for business monitoring.

Let´s first have a look at a typical message exchange for a service call:

Full Size
A Gliffy Diagram named: GeneratedEvents

Each service call generates four events two on the client side and two on the server side. The monitoring solution logs each of these events to a central database and also correlates the events so they can be viewed in the context of the original service call.

Architecture

The Talend Service Activity Monitoring soltion consists of three parts

  • Agent (sam-agent): Hooks into cxf on consumer and provider side to trace the messages sent and received. For each messages a monitorijng event is created and sent to the monitoring server
  • Monitoring Server (sam-server): Offers a Monitoring Service where it listens for incoming monitoring events. Each event is stored into the database. Offers a rest based interface to browse and search events
  • Monitoring GUI: Offers a web based user interface to browse and search monitoring events

While the Agent and Monitoring Server are part of the community edition and apache Licensed the Monitoring GUI will only be able for the Commercial product.

The following diagram shows how the components are deployed:

Full Size
A Gliffy Diagram named: Deployment

Event Structure in database

Field Type Description
ID bigint(20) Stores the persistence id after commit
MESSAGE_CONTENT longtext SOAP message content
EI_TIMESTAMP datetime Timestamp of the event creation
EI_EVENT_TYPE varchar EventType is an enumeration. Values: REQ_IN; REQ_OUT; RESP_IN; RESP_OUT; FAULT_IN; FAULT_OUT
ORIG_CUSTOM_ID varchar Custom id is an empty field. It is not used by Agents currently
ORIG_PROCESS_ID varchar Process id is the OS process id
ORIG_HOSTNAME varchar Hostname of the event creator
ORIG_IP varchar IP address of the event creator
MI_PORT_TYPE varchar Service port type of event creator
MI_OPERATION_NAME varchar Service operation name of event creator
MI_MESSAGE_ID varchar Unique id for one message. (UUID)
MI_FLOW_ID varchar Unique id for the message flow. (UUID) All events with the same id belong together
MI_TRANSPORT_TYPE varchar Transport type of event creator
ORIG_PRINCIPAL varchar Principal info in the message header
CONTENT_CUT tinyint Flag if event content was cut

Additionally to the fixed event structure the EVENTS_CUSTOMINFO table holds name/value pairs the customer can freely define.

Field Type Description
ID bigint id
EVENT_ID bigint ID of the event the custom info belongs to
CUST_KEY varchar custom property's key, e.g. Application name
CUST_VALUE varchar custom property's value, e.g. Dummy App

Filters and Handlers

On the Agent as well as on the Monitoring Server events can be filtered. An event filter is defined by an interface and decides if an event should be processed or discarded. There are already several filters available and new ones can be created by the customer by implementing the interface. Among the available filters the most powerful is a JXPath filter that can be used to filter events by any meta data. Handlers are processed after the filters and can change the events. Typical applications of handlers are hiding Passwords or Cutting off Attachments.

Try it yourself

You can download the Talend ESB Community Edition and try the Service Activity Monitroing yourself. The gettings started guide in the documentation will help with the setup.

http://de.talend.com/products-application-integration/application-integration-esb-se.php

Talend Releases new ESB product based on Apache projects

Talend recently released the new Talend ESB Standard Edition product that is based on many Apache projects (Apache CXF, Apache Camel, Apache Karaf). It allows users to build integrations and services using the reliable and wide spread Apache projects while extending these projects with several enterprise features:

  • Service Locator - provides automatic and transparent failover and load balancing between service Consumers and Providers through a newly developed extension that allows for dynamic endpoint registration and lookup via Apache Zookeeper
  • Service Activity Monitoring - facilitates the capture of analysis of service activity, including service response times, traffic patterns, auditing and more, by capturing events and storing information
  • Security Token Service (STS) Framework - supports Security Assertion Markup Language 2.0 (SAML 2.0) to federate security credentials. This allows clients and services to securely and transparently authenticate during connections without the need for custom coding

The product is Apache Licensed so users get great flexibility to use and even enhance the code.

While this already gives a lot of value to our customers there is much more to come in the near future. The big news is the unified platform that will come with the enterprise edition of the ESB. In the future all Talend products will use a common Development, Monitoring and Runtime Platform.

See also:

http://de.talend.com/products-application-integration/application-integration-esb-se.php

http://www.talend.com/press/Talend-Introduces-Industry-s-First-Unified-Platform-for-Data-Services-with-New-ESB-Offering.php

Labels: esb, camel, cxf, karaf, talend