[Message Prev][Message
Next][Thread Prev][Thread Next][Message
Index][Thread Index]
Re: xAP EOM identifier in xAP v1.3
Are you up for championing the config schema ? I am aware that work
is
being done elsewhere , at least covering BSC aspects.
I do like that approach, my only concern is that of keeping a hub as
lean and mean as possible which I believe is the approach xFX Express
takes. , and that really means no parsing of message content, although
maybe it could handle TCP connections differently.
K
Patrick Lidstone wrote:
>
>
> How about keeping it in-band (as xAP messages) but using it as the
> foundation for a generic configuration schema, which is looooong
> overdue! Would that work? There may be some benefit from other end
> points being aware of filtering changes etc.? Authentication would
> have to be a closed dialogue, but could still be xAP, between just the
> hub and client.
>
> (A tcp hub could just pass config schema messages and heartbeats by
> default on start up perhaps?)
>
> P.
>
> 2009/9/17 Kevin Hawkins <yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>
>
> If we wish to allow some inband mechansim for a TCP client to
> configure
> it's link eg for logon/authentication, selecting a preset profile
> governing which traffic is passed or to be able to create dynamic
> filters then we need some way to extract link control traffic.
Perhaps
> we could do this via a specific xAP schema targeted at the
hub/router
> (?) . The iServer currently uses a different
<stx><etx>
> combination for
> control and xAP traffic, rather than any escape sequence or auto
> detect
> of non xAP content. This allows mid session changes. The specifics
> could
> easily be changed as AFAIK there are only two people using the
dynamic
> filtering feature. It would be tody to allow the
> xRouter/xServer/iServer to be able to connect bidirectionally to a
> standard release hub.
>
> K
>
> Patrick Lidstone wrote:
> >
> >
> > Cool!
> >
> > I'm not sure the transport wrapper over TCP 'breaks
everything'
> - I've
> > used it for TCP routing of xAP over the internet framed using
the
> > transport wrapper for several years now (I even demoed it as
a means
> > of remotely accessing my home intranet via xAP at the
inaugural HA
> > weekend meet many moons ago), but perhaps I'm missing
something from
> > your requirements? All the tcp hub has to do is strip off the
start
> > and end delimiter before rebroadcast, and likewise frame any
> received
> > udp messages with an <stx>, <etx> if operating
bi-directionally.
> > Admittedly it's an additional (minimal) cost that isn't
present with
> > the double-0A scheme, but the benefits of being able to
detect the
> > start of frame, and the benefits of being able to apply this
> approach
> > to any async transport, not just tcp/ip, outweigh that in my
> mind. And
> > it is in the spec already ;-)
> >
> > Personally, I don't see a need for a delimiter at the end of
a
> message
> > in UDP xAP. I'm sure I'm going to be outvoted, but look at
all the
> > other UDP protocols out there - it's not common practice
because
> it's
> > not technically necessary, and as such it's just adding
unnecessary
> > cruft to xAP!
> >
> > Cheers
> > Patrick
> >
> >
> > 2009/9/17 Edward Pearson <edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>
> > <mailto:edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>>>
> >
> >
> >
> > You?re right, to get a connection in good time that?s not
> > statically configured it?d need a request/offer message
pair.
> >
> >
> >
> > The experimental hub I have here is designed to address
one
> > specific problem only ? unreliability over WiFi.
> >
> > In runs as a master slave arrangement. In master mode it
accepts
> > incoming tcp connections on port 3639. In slave mode
> (deployed on
> > the WiFi machine) it?s configured (statically) with the
IP
> address
> > or hostname of the master hub to which it connects over
tcp. The
> > tcp connection is used one-way only, from master to
slave. The
> > master forwards all xAP messages it receives (on udp) to
all its
> > local udp clients (as usual) and to any connected tcp
> clients. The
> > slave forwards all messages received on its tcp
connection
> to its
> > local clients on udp. Works a treat. Now I can lie on the
sofa
> > debugging Viewer v4 (it?ll be out really, really soon now
I
> > promise) on my WiFi laptop without wondering where 15% of
the
> > messages have gone.
> >
> >
> >
> > I mention all this mainly (and at risk of getting back to
> Kevin?s
> > original point of this thread which I?m feeling guilty of
> > hijacking horribly) because I needed to address the
problem
> of the
> > segregation of messages in a stream to get this to work.
So I
> > experimented with several schemes getting a good feel
for the
> > practical pro?s and cons of each. Initially I went for
the
> STX/ETX
> > method from the transport wrapper since it?s what?s in
the
> > existing spec (but opting out of the CRC). Later I used
the
> double
> > LF termination. The transport wrapper is somewhat easier
to work
> > with. For instance, it?s easier to re-sync to the a
message
> start
> > with transport wrapper; just wait for the next STX (a
single
> > byte), vs. needing match all of
?xap-[headerhbeat]<lf>?.
> Message
> > ends are similar: ETX (1 byte) vs <lf><lf> (2
bytes). But all
> > this is irrelevant in the udp world. Introducing STX/ETX
here
> > would break everything while double <lf> is nicely
backwards
> > compatible.
> >
> >
> >
> > (and finally, my point relevant to the thread start)
> >
> >
> >
> > I don?t think double <lf> is the right solution for
Kevin?s
> > developers? wireless app problem; it?d help them out
> somewhat but
> > I think it more than likely that it?s ?papering over? a
bug
> in or
> > around their network stack. But there is a valid need
lurking in
> > here; without message-level delimiting in the original
spec,
> > implementers of xAP commonly run up against the problem
of
> knowing
> > where the end of a message is while parsing messages. The
double
> > <lf> definitely helps here. So given that it?s easy
to add,
> helps
> > in some folks, eases adoption and causes no backwards
> > compatibility issues, I?m happy that it should go into
the
> v1.3 spec.
> >
> >
> >
> > *From:* xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>
> > [mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>]
*On Behalf Of *Patrick
> > Lidstone
> > *Sent:* 16 September 2009 12:00
> >
> > *To:* xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>
> > *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP
v1.3
> >
> >
> >
> >
> >
> > I agree that checksums are not appropriate to TCP
> connections, the
> > transport wrapper already designates them as optional.
> >
> > I don't think hubs currently send heartbeats? If they do,
> it's not
> > explicit in the spec. (Mine doesn't). We also need a
> mechanism to
> > differentiate between multiple hubs (since most people
will
> > generally run at least one hub per vm).
> >
> > Is a TCP hub going to relay all traffic to all TCP
> endpoints? That
> > won't scale very well... but it's also not clear how the
> filtering
> > etc. will work. Administering filters manually works fine
> for the
> > odd serial segment, but it's not going to be viable for a
large
> > number of endpoints.
> >
> > Heartbeats are broadcast every minute or so typically, so
that's
> > going to result in start up times of up to two minutes
for a tcp
> > end point. Is that acceptable? If not, do we reduce the
new tcp
> > hub heartbeat interval? Or is there a mechanism for an
> endpoint to
> > poke a hub and elicit a heartbeat?
> >
> > And lastly, is there merit in some kind of referrer
system to
> > allow for load-balancing and automated failover of tcp
> connections
> > across more than one hub? If so, how would this work?
> >
> > Patrick
> >
> > 2009/9/15 Edward Pearson <edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>
> > <mailto:edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>>>
> >
> >
> >
> > I don?t mean that packet fragmentation or re-ordering are
> myths ?
> > just that when dealing with xAP at the IP stack (socket)
> interface
> > you are dealing with datagrams not data packets so you
don?t
> need
> > to worry about those packet aspects. There was a design
decision
> > to limit xAP datagrams to 1500 bytes to improve the
likely
> > coverage of correctly sent datagrams in a world of IP
stacks of
> > varying quality. It?s no more important than that.
Readers
> of the
> > spec seem to attach more importance to it than it needs
(there?s
> > the myth aspect). To program xAP you don?t need to worry
about
> > fragmentation and reordering ? just keep your datagrams
1500
> bytes
> > or less and let the stack do its thing.
> >
> >
> >
> > Sometimes an app goes a bit wild (xAP news has
occasionally
> been a
> > useful test source) and big xAP messages are generated ?
and
> they
> > get delivered too! It?s normally the input buffer of the
> receiving
> > app (eg hub) that breaks first.
> >
> >
> >
> > On the tcp framing, I?d suggest that implementing the CRC
part
> > (irrelevant on a reliable stream) is a waste of most
people?s
> > time; by far more use will be made of tcp than any kind
of
> > serial/485/etc networks so they?d be sharing development
of an
> > implementation with nobody.
> >
> >
> >
> > Discovery can be done simply with the existing hub
heartbeat
> > message. Just need to agree on an extra block that
> advertises the
> > port number of the tcp service ? which I assume, by
default
> would
> > be 3639.
> >
> >
> >
> > Having read the 802.11 spec, I now understand why
broadcast udp
> > from an AP to a NIC is so un-reliable. And ad-hoc mode is
a real
> > disaster!
> >
> >
> >
> > *From:* xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>
> > [mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>]
*On Behalf Of *Patrick
> > Lidstone
> > *Sent:* 15 September 2009 14:15
> >
> >
> > *To:* xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>
> > *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP
v1.3
> >
> >
> >
> >
> >
> > Hi Edward,
> > Please see in-line responses...
> >
> > 2009/9/15 Edward Pearson <edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>
> > <mailto:edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>>>
> >
> >
> >
> > The double 0x0a terminator works for me, it?s simple to
> implement
> > ? already done for my stuff.
> >
> >
> >
> > ...but it isn't robust unless the message is sent as a
single
> > datagram, and if it is sent as a single datagram, the os
should
> > deliver it as such -- and if it doesn't, adding an eom
marker
> > isn't going to help.
> >
> > For reliable streams, such as TCP, I generally frame
the
> > messages with STX and ETX.
> >
> >
> >
> > I thought all this business about 1500 bytes was
somewhat
> > urban myth (at least as it applies to xAP). The data
packet
> > size is not what?s important; it?s how the particular
IP
> stack
> > implementation deals with >datagrams< that?s
the key.
> >
> >
> > I don't understand what you are syaing here? The data
packet
> size
> > is inextricably linked to the IP stack. What aspect is it
> that you
> > consider to be an urban myth?
> >
> >
> > I only have experience of the two most recent Windows
stacks
> > (XP and Vista) which I agree are likely more capable
> than old
> > embedded stacks. My experiments with Wireshark show
that
> those
> > stacks will happily deal with fragmentation and
> > defragmentation. 64KB in a single datagram? No
problem
> if your
> > receive buffer is long enough (even if you force a
small
> MTU).
> >
> > Yes, agreed, OSes perform fragmentation for you. The
individual
> > fragments have a maximum size as determined by the MTU.
For
> > pragmatic, practical reasons, xAP needs to define a
maximum
> > overall message size, and for convenience's sake that was
set as
> > equal to the 'standard' MTU size. Devices which use a
> smaller MTU
> > /should /fragment and reassemble seamlessly provided that
the
> > correct socket options have been set to define the
maximum UDP
> > packet size. By electing to use a single MTU's worth of
data, we
> > we avoid the overhead associated with fragmentation and
> reassembly
> > (principally memory buffering) which is a good thing.
When
> > reassembled, the os should deliver a datagram as a
complete
> entity
> > in one go (irrespective of the mtu size, assuming that
the
> > datagram falls within the maximum datagram size defined
for the
> > stack). If the sender doesn't send a message as a single
> datagram,
> > then the whole thing falls apart because effectively you
are
> then
> > doing fragmentation and reassembly at the application
layer, and
> > that won't work because the ordering across datagrams is
not
> > guaranteed and individual datagrams may get lost.
> >
> >
> > If anything goes wrong (eg, missing fragment) the
whole
> > datagram is dropped. I can?t see how packet
re-ordering can
> > happen on a single LAN for broadcast UDP ? there are
no
> > multiple paths and no retry mechanism. Certainly
never
> > observed it ? I assume the stack would again just
drop the
> > entire datagram.
> >
> >
> > Re-ordering of datagrams can occur for multiple reasons
on a
> > single lan. A sender may send individual UDP packets in
any
> order
> > it chooses. This commonly occurs with fragmented packets
> > originating from a multi-threaded sender, which can be
> interleaved
> > with smaller, non-fragmented datagrams as required
(optimising
> > transient memory use, as soon as they are sent, the
buffer
> can be
> > released). A switch is not obliged to retransmit packets
in the
> > order in which they are received. And most fundamentally,
the
> > specs do not require UDP packets to be ordered so, even
if you
> > don't observe it, it /can/ happen, and sooner or later
> > interoperability issues will arise if the working
assumption is
> > made that they always arrive in order.
> >
> >
> >
> > A bigger issue for me, and the reason for me
experimenting a
> > few times with TCP stream serving from hubs, is
datagram
> loss
> > over WiFi networks. This is greatly accentuated for
UDP when
> > you use broadcast (as we do) from wired to wireless
> (fine the
> > other way as it?s not really a broadcast packet till
it gets
> > to the AP) as the access point and NIC can not longer
apply
> > their ack/nak procedure at the transport level. I
> commonly see
> > xAP datagram loss rates from wired to wireless
> connections of
> > 20%. So I?d like us to agree on a standard transport
wrapper
> > for TCP streams which a lot of platforms would find
useful.
> >
> >
> >
> > I'd suggest using the same framing as the "transport
> wrapper", as
> > this then allows for code to be shared across transports.
If xAP
> > was extended to support TCP, then that should also
include a
> > formal discovery mechanism by which the IP
> address/characteristics
> > of the hub can be discovered (over UDP xAP?)
> >
> > Patrick
> >
> >
> > *From:* xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>
> > [mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>]
*On Behalf Of *Patrick
> > Lidstone
> > *Sent:* 14 September 2009 14:19
> > *To:* xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>
> > <mailto:xAP_developer@xxxxxxx
> <mailto:xAP_developer@xxxxxxx>>
> > *Subject:* Re: [xAP_developer] xAP EOM identifier in
xAP
> v1.3
> >
> >
> >
> >
> >
> > So the xAP delimiters for serial are defined in the
1.2 xAP
> > spec here:
> >
> http://www.xapautomation.org/index.php?title=Protocol_definition#Transport_Wrapper
> >
> > To the best of my knowledge, the MTU size for an
ethernet
> > frame is 1518 bytes, which leads to a UDP packet MTU
of 1500
> > bytes and this is the size that is adopted by the
> majority of
> > operating systems. Internet routers (i.e. ISPs)
> sometimes use
> > an MTU of 576 bytes, but this wouldn't be relevant to
xAP
> > since the traffic doesn't pass over the wider net, or
if it
> > does, it's generally gatewayed via a TCP/IP
connection.
> >
> > If a device is receiving fragmented UDP packets, I
think the
> > same issues arise as those related to extending the
xAP
> > message size beyond 1500 bytes - what happens if a
fragment
> > gets discarded.
> >
> > If you take the scenario:
> >
> > Part 1(a), Part 1(b), Part 2(a), Part 2(b), Part
3(a),
> Part 3(b)
> >
> > --- first there is no guarantee that the parts will
be
> > delivered in order, and second, if part 1(b) was
> dropped, and
> > you were blindly assembling messages based on the
proposed
> > double-0a terminator, you'd end up with a message
comprising
> > part 1(a), 2(a) and 2(b) which is not only obviously
> corrupt,
> > but also possibly larger than the maximum xAP message
size,
> > blowing away your buffers.
> >
> > I think the solution is to probably parse incoming
> messages on
> > the fly, byte-by-byte. You can then at least reset
your
> state
> > when you encounter the xap-header, and if you count
open and
> > close curly braces, you can tell when you have an
apparently
> > complete message. This won't solve the issue of UDP
> fragments
> > being potentially received out of order, but so long
as
> we are
> > dealing with a single LAN, and fragmentation occurs
at the
> > receiver, we will be ok I think.
> >
> > It is absolutely possible for UDP packets to be
> discarded, and
> > the way we deal with this in xAP is to accept that
this can
> > happen to xAP messages, and layer application level
> > acknowledgements where knowing that a message has
been
> > received is critically important - whether explicitly
or
> > implicitly through status messages. There are various
> schemes
> > that could be adopted to allow a receiver to detect
lost
> > messages (e.g. sequence numbering), but they quickly
become
> > quite onerous, and assume that the originator keeps a
> copy of
> > the original message or is able to reconstruct it -
which is
> > non-trivial for the many xAP nodes.
> >
> > Perhaps you can share more of the specific details of
the
> > device(s) in question (manufacturer, docs, o/s etc),
and
> their
> > specific behaviour, which seems a bit anomalous?
> >
> > Patrick
> >
> > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>>
> >
> > One of the issues seems to be that there is
conflicting
> views
> > as to the
> > length of a a UDP data packet payload. Some people
cite 500
> > or 512
> > characters and some 1500. Regardless in some low
> > memory/performance
> > devices it is being reported, that even with UDP,
> packets are
> > being
> > received from the buffer either truncated or appended
> back to
> > back.
> > The latter I assume is due to the speed of the device
in
> > servicing the
> > buffer.
> >
> > We have an opportunity to protect against this with
v1.3 and
> > the double
> > '0A' seems the most compatible. I would be loathe
to
> support
> > anything
> > that wasn't backward compatible.
> >
> > K
> >
> >
> > Patrick Lidstone wrote:
> > >
> > >
> > > I will dig it out - it included an optional
checksum I
> > think, and IIRC
> > > was framed by stx and etx (a kind of pseudo
industry
> > standard). I
> > > certainly used it with the PIC serial stuff and
the
> > xAP-serial bridge.
> > > Re.: long message truncation and concatenation:
If we need
> > to support
> > > messages that are larger than one UDP packet,
then
> there are
> > > additional complexities and the proposed scheme
won't work
> > as intended
> > > as the ordering of UDP messages is not
guaranteed. I'm
> happy
> > to help
> > > refine a spec to support these capabilities, but
it is
> > moving away
> > > from the basic ethos of xAP somewhat, as devices
would
> have
> > to be able
> > > to buffer received messages, and that raises the
bar
> > considerably.
> > > Perhaps there is scope for co-existence of a
long and
> > standard message
> > > protocol though?
> > >
> > > Patrick
> > >
> > > 2009/9/13 Kevin Hawkins
<yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>
> >
> > > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>>>
> >
> > >
> > > ... Oh ... where is that in the spec ? it
might
> be all
> > we need.
> > >
> > > This is also tied in with some aspects of
long
> > message truncation
> > > and concatenation of messages received in
UDP receive
> > buffers
> > > though...
> > >
> > > K
> > >
> > > Patrick Lidstone wrote:
> > > >
> > > >
> > > > The original xap spec provides
extensions for
> framing
> > a message over
> > > > async serial which also delimit the
start of the
> > message - you don't
> > > > need this 'hack' if you follow the
original spec for
> > non-UDP
> > > transports.
> > > >
> > > > Patrick
> > > >
> > > > 2009/9/13 Kevin Hawkins
> <yahoogroupskh@xxxxxxx <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>
> > > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>>
> >
> > > > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>
> >
> > > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>
> > <mailto:yahoogroupskh@xxxxxxx
> <mailto:yahoogroupskh@xxxxxxx>>>>>
> > > >
> > > > We have been asked on several
occasions how to
> > detect the
> > > end of a
> > > > xAP messager as there is no unique
EOM
> character.
> > Typically
> > > in any
> > > > reasonable sized packet structured
transport eg
> > UDP then the
> > > packet
> > > > provides such an indication but on
systems with
> > small packets or
> > > > non eg
> > > > asynchronous serial this is not
useable.
> > > >
> > > > In discussing this with the
specification team
> > we must
> > > consider
> > > > backwards compatability and so we
do not wish to
> > alter the
> > > > specification
> > > > to include a unique EOM character.
What we
> do propose
> > > however is that
> > > > xAP v1.3 will specify that all
messages
> should end
> > with two
> > > > consecutive
> > > > chr(10)'s immediately after the
closing '}'
> > > >
> > > > ie ..... 7D 0A 0A
> > > >
> > > > Some apps, even v1.2 ones,
already do
> this. We
> > don't
> > > believe this
> > > > will cause any backwards
compatibility
> issues and
> > it will
> > > always be
> > > > unique within a xAP message.
> > > >
> > > > So, in developing xAP v1.3 apps
could you
> > therefore append
> > > two 0A's
> > > > at the end of your messages , and
of course
> handle
> > incoming
> > > messages
> > > > containing such.
> > > >
> > > > K
> > > >
> > > >
> > > >
------------------------------------
> > > >
> > > > Yahoo! Groups Links
> > > >
> > > >
> > > >
> > mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>>
> > > >
> <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>>>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > ------------------------------------
> > >
> > > Yahoo! Groups Links
> > >
> > >
> > >
> mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>>
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> > mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > <mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
> mailto:xAP_developer-fullfeatured@xxxxxxx
> <mailto:xAP_developer-fullfeatured@xxxxxxx>
>
>
>
>
>
>
------------------------------------
xAP_Development Main Index |
xAP_Development Thread Index |
xAP_Development Home |
Archives Home
|