Re: Message Queuing

[Stuart Booth <lists@xxxxxxxxxxxxxxxx>]

Almost a month on, whoops. Summery weather. Bahhh.

On Mon, 27 Jun 2005 17:41:10 -0400, "Sullivan, Glenn"
<gsullivan@xxxxxxx> wrote:

>What do you do in viewer?  Viewer seems to catch everything, and
frankly, I've noticed that I get almost NO dropped messages when viewer
isn't running...

Viewer's not so great at ordering all the messages in memory. It
retains everything so that you can sift through things and show
subsets of information. As messages arrive Viewer does many things
with them to determine if they're relevant for the view currently
being shown. You can turn some categories of messages on/off in the
tree at different levels which make it slightly more complicated. All
this processing takes a bit of time, especially under heavy incoming
load.

The latest version is somewhat better at this but the grid view is
less than efficient at this time (hark, did somebody call out "ToDo
List!" at the back there?)...

>Regardless, the problem still exists.  Too many messages for everything
to process at once.  I'm going to start queuing inbound messages I think...

I use a threaded message queue in many of my xAPplications
(SlimConnector, TiVoYAC, MCxAP, to name a few off the top of my head).
This evening I've added that same queue deep into xFx. This means that
all incoming xAP data gets queued up so that listening for new events
can continue as quickly as possible. Which should mean no dropped
messages.

A separate thread processes messages as they appear in the queue. This
is the decoding of the raw xAP data into an xFx object, checking for
errors, and firing off any events that your xAPplication may install.
All the complicated stuff in other words.

I've just been doing some testing with my xAP Message Flood utility to
generate huge xAP messages volumes. On my dev. box at least (with a
Debug Viewer running under the debugger, atop my Service-based Hub)
the message queue only starts to backlog after about 3000 xAP
messages/minute (that Flood is attempting to pump out). Of course this
is based on a pretty simple set of 26 messages for sending, but one of
them must be BSC related as it's causing a xAPplication on my Server
to pump out a lot of BSC status information messages in response.

Around the 2500 messages/minute mark and upwards, the queue burps
occasionally but quickly empties - ie Viewer is keeping up. Around the
3000 mark the queue starts to fill slowly - usually after something
burps out a lot of messages perhaps. Since I never see that kind of
message volume normally, that'll be why I've never seen this causing
any problems before.

Anyway, this is now built directly into xFx so all xFx-based
applications will benefit without changes required.

S
--
Stuart Booth <stuart@xxxxxxx>
xAPFramework.NET - a xAP software development framework for .NET

http://www.xapautomation.org/ 
     http://www.xapframework.net/