Event Distribution Performance Tests

There are fundamental differences between the TAO event service and IceStorm. The most important difference is that the TAO event service uses the CORBA type Any for forwarding event data, while IceStorm delivers events by calling an operation with parameters. The former is called an untyped event service, the latter is called a typed event service.

Note that CORBA also defines a typed event service, but there are very few implementations. In particular, TAO does not have a typed event service. To quote from the TAO FAQ:

Q: Does TAO's COS Event Service support Typed Events?
A: No, and there are no plans to do so. To quote Carlos O'Ryan (coryan@uci.edu): "The spec is so loose that it is hard to define what an implementation would be, and implementations probably won't be interoperable."

We believe that typed event services are far superior to untyped ones, not only with respect to performance (using the type Any to forward events brings with it a lot of overhead), but also with respect to ease of use. Unfortunately, CORBA makes the implementation of a typed event service very difficult, due to the lack of APIs for simple and efficient request forwarding. Even if ORB-internal APIs were to be used, efficient request forwarding would be difficult or impossible for older IIOP protocol versions or if IIOP code set negotiation is needed.

With Ice, on the other hand, request forwarding is very simple, supported by both a straightforward protocol and easy-to-use APIs. For more information on how to forward requests in Ice as "blobs", and how to use ZeroC's typed event service, please consult the Ice Manual.

Test Setup

Tests were performed on our Linux system as described in the general test setup. Event publishers and subscribers were configured as for all other tests, that is, we used a thread pool model with four worker threads. In all tests, publishers sent events to the event service using twoway invocations.

For the TAO event service in unbuffered mode, we used the following configuration:

static CEC_Factory "-CECDispatching reactive"

We chose this configuration because it was the fastest in our tests, and we wanted to use whatever configuration is optimal for the TAO event service. (In contrast to the client and server configurations, the user should not be concerned with the concurrency model used by middleware services, as long as the services perform their task.)

For buffered mode, we used:

static CEC_Factory "-CECDispatching mt"

IceStorm was configured to use the thread pool model, with a single worker thread (which is the default configuration).

Event Service Latency

Our first test compared the latency of IceStorm and the TAO event service. We measured latency by sending events at regular intervals. Each event contains the time at which it was created, so that the event subscriber can calculate the latency as the difference of the current time and the time of the event creation.

To get comparable results, both IceStorm and the TAO event service used unbuffered twoway delivery mode. Events per second were calculated as the number of subscribers divided by the time per event.

Event Service Latency
Unbuffered Twoway

Publishers- Subscribers	IceStorm Time per Event	TAO Event Service Time per Event	IceStorm Events per Second	TAO Event Service Events per Second	Difference
1 - 1	0.146 ms	0.420 ms	6845	2378	188%
1 - 2	0.199 ms	0.666 ms	10036	3005	234%
1 - 5	0.390 ms	1.339 ms	12831	3735	244%
1 - 10	0.728 ms	2.542 ms	13738	3933	249%
1 - 20	1.401 ms	5.201 ms	14275	3846	271%

As the data shows, the latency for sending events with IceStorm is 2.8 to 3.7 times better than with the TAO event service, depending on the number of subscribers.

Event Service Throughput

More important than event service latency is event service throughput. To measure event throughput with IceStorm, we used the following typed event tick:

enum AEnum
{
    A, B, C
};

struct AStruct
{
    AEnum e;
    string s;
    double d;
};

interface Ping
{
    void tick(long timestamp, AEnum e, int i, AStruct s);
};

For the TAO event service, we used untyped events. A struct Event was stored in an Any:

enum AEnum
{
    A, B, C
};

struct AStruct
{
    AEnum e;
    string s;
    double d;
};

struct Event
{
   long long timestamp;
   AEnum e;
   int i;
   AStruct s;
};

We measured throughput by sending events as quickly as possible. Subscribers record the times at which the first and the last event are received. The number of events divided by the difference of these two times provides the number of events per second. The table shows the mean events per second for all subscribers.

In our first test, we used unbuffered twoway delivery for both IceStorm and the TAO event service:

Event Service Throughput
Unbuffered Twoway

Publishers- Subscribers	IceStorm Time per Event	TAO Event Service Time per Event	IceStorm Events per Second	TAO Event Service Events per Second	Difference
1 - 1	0.198 ms	0.768 ms	5043	1301	288%
1 - 10	1.176 ms	4.732 ms	8501	2113	302%
10 - 1	0.226 ms	0.728 ms	4426	1375	222%
5 - 5	0.636 ms	2.478 ms	7858	2018	289%

As for event latency, there is a large performance gap between the TAO event service and IceStorm. IceStorm can deliver 3.2 to 4 times as many events than the TAO event service using unbuffered twoway delivery.

Our next test used buffered event forwarding. In contrast to the tests above, IceStorm and the TAO event service do not forward events as they receive them; instead, they first buffer the events and use service-internal threads to send the events to the subscribers. (This delivery model has the advantage that a misbehaving publisher or subscriber cannot block the event forwarding service.)

There is one big difference between the TAO event service and IceStorm for this delivery model: IceStorm uses batched oneway delivery, which inherently provides buffering, while TAO uses regular twoways and service-internal mechanisms to store and forward events.

Event Service Throughput
Batched Oneways / Buffered Twoway

Publishers- Subscribers	IceStorm Time per Event	TAO Event Service Time per Event	IceStorm Events per Second	TAO Event Service Events per Second	Difference
1 - 1	0.123 ms	0.847 ms	8125	1181	588%
1 - 10	1.046 ms	4.834 ms	9561	2069	362%
10 - 1	0.129 ms	0.562 ms	7759	1779	336%
5 - 5	0.492 ms	2.120 ms	10162	2359	331%

Once again, the data shows the design advantages of Ice's batch mode combined with efficient forwarding mechanisms: IceStorm can deliver up to 6.8 times as many events as the TAO event service with a comparable delivery model.

 

(Return to the Performance Test Index)

Message Forwarding

Nearly all non-trivial distributed applications require fast message forwarding. Message forwarding can take many different forms. One common message forwarding scenario are clients that need to access a server via a router, such as a firewall, and receive callbacks via the same router. The probably most common message forwarding mechanism are event publishers that send messages to event subscribers. An event service decouples publishers from subscribers, and forwards the events to subscribers. For our event distribution tests, we used the latter scenario and compared the performance of the TAO implementation of the CORBA event service (CosEvent) with IceStorm.