[Message Prev][Message
Next][Thread Prev][Thread Next][Message
Index][Thread Index]
Re: xAP EOM identifier in xAP v1.3
--00163600ce79dd64220473c36751
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
How about keeping it in-band (as xAP messages) but using it as the
foundation for a generic configuration schema, which is looooong overdue!
Would that work? There may be some benefit from other end points being
awar=
e
of filtering changes etc.? Authentication would have to be a closed
dialogue, but could still be xAP, between just the hub and client.
(A tcp hub could just pass config schema messages and heartbeats by default
on start up perhaps?)
P.
2009/9/17 Kevin Hawkins <yahoogroupskh@xxxxxxx>
> If we wish to allow some inband mechansim for a TCP client to
configure
> it's link eg for logon/authentication, selecting a preset profile
> governing which traffic is passed or to be able to create dynamic
> filters then we need some way to extract link control traffic. Perhaps
> we could do this via a specific xAP schema targeted at the hub/router
> (?) . The iServer currently uses a different <stx><etx>
combination for
> control and xAP traffic, rather than any escape sequence or auto
detect
> of non xAP content. This allows mid session changes. The specifics
could
> easily be changed as AFAIK there are only two people using the dynamic
> filtering feature. It would be tody to allow the
> xRouter/xServer/iServer to be able to connect bidirectionally to a
> standard release hub.
>
> K
>
> Patrick Lidstone wrote:
> >
> >
> > Cool!
> >
> > I'm not sure the transport wrapper over TCP 'breaks everything' -
I've
> > used it for TCP routing of xAP over the internet framed using the
> > transport wrapper for several years now (I even demoed it as a
means
> > of remotely accessing my home intranet via xAP at the inaugural
HA
> > weekend meet many moons ago), but perhaps I'm missing something
from
> > your requirements? All the tcp hub has to do is strip off the
start
> > and end delimiter before rebroadcast, and likewise frame any
received
> > udp messages with an <stx>, <etx> if operating
bi-directionally.
> > Admittedly it's an additional (minimal) cost that isn't present
with
> > the double-0A scheme, but the benefits of being able to detect
the
> > start of frame, and the benefits of being able to apply this
approach
> > to any async transport, not just tcp/ip, outweigh that in my
mind. And
> > it is in the spec already ;-)
> >
> > Personally, I don't see a need for a delimiter at the end of a
message
> > in UDP xAP. I'm sure I'm going to be outvoted, but look at all
the
> > other UDP protocols out there - it's not common practice because
it's
> > not technically necessary, and as such it's just adding
unnecessary
> > cruft to xAP!
> >
> > Cheers
> > Patrick
> >
> >
> > 2009/9/17 Edward Pearson <edward.mailgroup@xxxxxxx
> > <mailto:edward.mailgroup@xxxxxxx>>
> >
> >
> >
> > You=92re right, to get a connection in good time that=92s not
> > statically configured it=92d need a request/offer message
pair.
> >
> >
> >
> > The experimental hub I have here is designed to address one
> > specific problem only =96 unreliability over WiFi.
> >
> > In runs as a master slave arrangement. In master mode it
accepts
> > incoming tcp connections on port 3639. In slave mode
(deployed on
> > the WiFi machine) it=92s configured (statically) with the IP
addres=
s
> > or hostname of the master hub to which it connects over tcp.
The
> > tcp connection is used one-way only, from master to slave.
The
> > master forwards all xAP messages it receives (on udp) to all
its
> > local udp clients (as usual) and to any connected tcp
clients. The
> > slave forwards all messages received on its tcp connection to
its
> > local clients on udp. Works a treat. Now I can lie on the
sofa
> > debugging Viewer v4 (it=92ll be out really, really soon now I
> > promise) on my WiFi laptop without wondering where 15% of the
> > messages have gone.
> >
> >
> >
> > I mention all this mainly (and at risk of getting back to
Kevin=92s
> > original point of this thread which I=92m feeling guilty of
> > hijacking horribly) because I needed to address the problem
of the
> > segregation of messages in a stream to get this to work. So
I
> > experimented with several schemes getting a good feel for
the
> > practical pro=92s and cons of each. Initially I went for the
STX/ET=
X
> > method from the transport wrapper since it=92s what=92s in
the
> > existing spec (but opting out of the CRC). Later I used the
double
> > LF termination. The transport wrapper is somewhat easier to
work
> > with. For instance, it=92s easier to re-sync to the a message
start
> > with transport wrapper; just wait for the next STX (a single
> > byte), vs. needing match all of
=93xap-[headerhbeat]<lf>=94. Messa=
ge
> > ends are similar: ETX (1 byte) vs <lf><lf> (2
bytes). But all
> > this is irrelevant in the udp world. Introducing STX/ETX here
> > would break everything while double <lf> is nicely
backwards
> > compatible.
> >
> >
> >
> > (and finally, my point relevant to the thread start)
> >
> >
> >
> > I don=92t think double <lf> is the right solution for
Kevin=92s
> > developers=92 wireless app problem; it=92d help them out
somewhat b=
ut
> > I think it more than likely that it=92s =91papering over=92 a
bug i=
n or
> > around their network stack. But there is a valid need lurking
in
> > here; without message-level delimiting in the original spec,
> > implementers of xAP commonly run up against the problem of
knowing
> > where the end of a message is while parsing messages. The
double
> > <lf> definitely helps here. So given that it=92s easy
to add, helps
> > in some folks, eases adoption and causes no backwards
> > compatibility issues, I=92m happy that it should go into the
v1.3 s=
pec.
> >
> >
> >
> > *From:* xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>
> > [mailto:xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>]
*On Behalf Of *Patrick
> > Lidstone
> > *Sent:* 16 September 2009 12:00
> >
> > *To:* xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>
> > *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP v1.3
> >
> >
> >
> >
> >
> > I agree that checksums are not appropriate to TCP
connections, the
> > transport wrapper already designates them as optional.
> >
> > I don't think hubs currently send heartbeats? If they do,
it's not
> > explicit in the spec. (Mine doesn't). We also need a
mechanism to
> > differentiate between multiple hubs (since most people will
> > generally run at least one hub per vm).
> >
> > Is a TCP hub going to relay all traffic to all TCP endpoints?
That
> > won't scale very well... but it's also not clear how the
filtering
> > etc. will work. Administering filters manually works fine for
the
> > odd serial segment, but it's not going to be viable for a
large
> > number of endpoints.
> >
> > Heartbeats are broadcast every minute or so typically, so
that's
> > going to result in start up times of up to two minutes for a
tcp
> > end point. Is that acceptable? If not, do we reduce the new
tcp
> > hub heartbeat interval? Or is there a mechanism for an
endpoint to
> > poke a hub and elicit a heartbeat?
> >
> > And lastly, is there merit in some kind of referrer system to
> > allow for load-balancing and automated failover of tcp
connections
> > across more than one hub? If so, how would this work?
> >
> > Patrick
> >
> > 2009/9/15 Edward Pearson <edward.mailgroup@xxxxxxx
> > <mailto:edward.mailgroup@xxxxxxx>>
> >
> >
> >
> > I don=92t mean that packet fragmentation or re-ordering are
myths =
=96
> > just that when dealing with xAP at the IP stack (socket)
interface
> > you are dealing with datagrams not data packets so you
don=92t need
> > to worry about those packet aspects. There was a design
decision
> > to limit xAP datagrams to 1500 bytes to improve the likely
> > coverage of correctly sent datagrams in a world of IP stacks
of
> > varying quality. It=92s no more important than that. Readers
of the
> > spec seem to attach more importance to it than it needs
(there=92s
> > the myth aspect). To program xAP you don=92t need to worry
about
> > fragmentation and reordering =96 just keep your datagrams
1500 byte=
s
> > or less and let the stack do its thing.
> >
> >
> >
> > Sometimes an app goes a bit wild (xAP news has occasionally
been a
> > useful test source) and big xAP messages are generated =96
and they
> > get delivered too! It=92s normally the input buffer of the
receivin=
g
> > app (eg hub) that breaks first.
> >
> >
> >
> > On the tcp framing, I=92d suggest that implementing the CRC
part
> > (irrelevant on a reliable stream) is a waste of most
people=92s
> > time; by far more use will be made of tcp than any kind of
> > serial/485/etc networks so they=92d be sharing development of
an
> > implementation with nobody.
> >
> >
> >
> > Discovery can be done simply with the existing hub heartbeat
> > message. Just need to agree on an extra block that advertises
the
> > port number of the tcp service =96 which I assume, by default
would
> > be 3639.
> >
> >
> >
> > Having read the 802.11 spec, I now understand why broadcast
udp
> > from an AP to a NIC is so un-reliable. And ad-hoc mode is a
real
> > disaster!
> >
> >
> >
> > *From:* xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>
> > [mailto:xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>]
*On Behalf Of *Patrick
> > Lidstone
> > *Sent:* 15 September 2009 14:15
> >
> >
> > *To:* xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>
> > *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP v1.3
> >
> >
> >
> >
> >
> > Hi Edward,
> > Please see in-line responses...
> >
> > 2009/9/15 Edward Pearson <edward.mailgroup@xxxxxxx
> > <mailto:edward.mailgroup@xxxxxxx>>
> >
> >
> >
> > The double 0x0a terminator works for me, it=92s simple to
implement
> > =96 already done for my stuff.
> >
> >
> >
> > ...but it isn't robust unless the message is sent as a single
> > datagram, and if it is sent as a single datagram, the os
should
> > deliver it as such -- and if it doesn't, adding an eom marker
> > isn't going to help.
> >
> > For reliable streams, such as TCP, I generally frame the
> > messages with STX and ETX.
> >
> >
> >
> > I thought all this business about 1500 bytes was somewhat
> > urban myth (at least as it applies to xAP). The data
packet
> > size is not what=92s important; it=92s how the particular
IP st=
ack
> > implementation deals with >datagrams< that=92s the
key.
> >
> >
> > I don't understand what you are syaing here? The data packet
size
> > is inextricably linked to the IP stack. What aspect is it
that you
> > consider to be an urban myth?
> >
> >
> > I only have experience of the two most recent Windows
stacks
> > (XP and Vista) which I agree are likely more capable than
old
> > embedded stacks. My experiments with Wireshark show that
those
> > stacks will happily deal with fragmentation and
> > defragmentation. 64KB in a single datagram? No problem if
your
> > receive buffer is long enough (even if you force a small
MTU).
> >
> > Yes, agreed, OSes perform fragmentation for you. The
individual
> > fragments have a maximum size as determined by the MTU. For
> > pragmatic, practical reasons, xAP needs to define a maximum
> > overall message size, and for convenience's sake that was set
as
> > equal to the 'standard' MTU size. Devices which use a smaller
MTU
> > /should /fragment and reassemble seamlessly provided that the
> > correct socket options have been set to define the maximum
UDP
> > packet size. By electing to use a single MTU's worth of data,
we
> > we avoid the overhead associated with fragmentation and
reassembly
> > (principally memory buffering) which is a good thing. When
> > reassembled, the os should deliver a datagram as a complete
entity
> > in one go (irrespective of the mtu size, assuming that the
> > datagram falls within the maximum datagram size defined for
the
> > stack). If the sender doesn't send a message as a single
datagram,
> > then the whole thing falls apart because effectively you are
then
> > doing fragmentation and reassembly at the application layer,
and
> > that won't work because the ordering across datagrams is not
> > guaranteed and individual datagrams may get lost.
> >
> >
> > If anything goes wrong (eg, missing fragment) the whole
> > datagram is dropped. I can=92t see how packet re-ordering
can
> > happen on a single LAN for broadcast UDP =96 there are no
> > multiple paths and no retry mechanism. Certainly never
> > observed it =96 I assume the stack would again just drop
the
> > entire datagram.
> >
> >
> > Re-ordering of datagrams can occur for multiple reasons on a
> > single lan. A sender may send individual UDP packets in any
order
> > it chooses. This commonly occurs with fragmented packets
> > originating from a multi-threaded sender, which can be
interleaved
> > with smaller, non-fragmented datagrams as required
(optimising
> > transient memory use, as soon as they are sent, the buffer
can be
> > released). A switch is not obliged to retransmit packets in
the
> > order in which they are received. And most fundamentally, the
> > specs do not require UDP packets to be ordered so, even if
you
> > don't observe it, it /can/ happen, and sooner or later
> > interoperability issues will arise if the working assumption
is
> > made that they always arrive in order.
> >
> >
> >
> > A bigger issue for me, and the reason for me
experimenting a
> > few times with TCP stream serving from hubs, is datagram
loss
> > over WiFi networks. This is greatly accentuated for UDP
when
> > you use broadcast (as we do) from wired to wireless (fine
the
> > other way as it=92s not really a broadcast packet till it
gets
> > to the AP) as the access point and NIC can not longer
apply
> > their ack/nak procedure at the transport level. I
commonly see
> > xAP datagram loss rates from wired to wireless
connections of
> > 20%. So I=92d like us to agree on a standard transport
wrapper
> > for TCP streams which a lot of platforms would find
useful.
> >
> >
> >
> > I'd suggest using the same framing as the "transport
wrapper", as
> > this then allows for code to be shared across transports. If
xAP
> > was extended to support TCP, then that should also include a
> > formal discovery mechanism by which the IP
address/characteristics
> > of the hub can be discovered (over UDP xAP?)
> >
> > Patrick
> >
> >
> > *From:* xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>
> > [mailto:xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>]
*On Behalf Of *Patrick
> > Lidstone
> > *Sent:* 14 September 2009 14:19
> > *To:* xAP_developer@xxxxxxx
> > <mailto:xAP_developer@xxxxxxx>
> > *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP
v1.3
> >
> >
> >
> >
> >
> > So the xAP delimiters for serial are defined in the 1.2
xAP
> > spec here:
> >
> http://www.xapautomation.org/index.php?title=3DProtocol_definition#Transp=
ort_Wrapper
> >
> > To the best of my knowledge, the MTU size for an ethernet
> > frame is 1518 bytes, which leads to a UDP packet MTU of
1500
> > bytes and this is the size that is adopted by the
majority of
> > operating systems. Internet routers (i.e. ISPs) sometimes
use
> > an MTU of 576 bytes, but this wouldn't be relevant to xAP
> > since the traffic doesn't pass over the wider net, or if
it
> > does, it's generally gatewayed via a TCP/IP connection.
> >
> > If a device is receiving fragmented UDP packets, I think
the
> > same issues arise as those related to extending the xAP
> > message size beyond 1500 bytes - what happens if a
fragment
> > gets discarded.
> >
> > If you take the scenario:
> >
> > Part 1(a), Part 1(b), Part 2(a), Part 2(b), Part 3(a),
Part 3(b=
)
> >
> > --- first there is no guarantee that the parts will be
> > delivered in order, and second, if part 1(b) was dropped,
and
> > you were blindly assembling messages based on the
proposed
> > double-0a terminator, you'd end up with a message
comprising
> > part 1(a), 2(a) and 2(b) which is not only obviously
corrupt,
> > but also possibly larger than the maximum xAP message
size,
> > blowing away your buffers.
> >
> > I think the solution is to probably parse incoming
messages on
> > the fly, byte-by-byte. You can then at least reset your
state
> > when you encounter the xap-header, and if you count open
and
> > close curly braces, you can tell when you have an
apparently
> > complete message. This won't solve the issue of UDP
fragments
> > being potentially received out of order, but so long as
we are
> > dealing with a single LAN, and fragmentation occurs at
the
> > receiver, we will be ok I think.
> >
> > It is absolutely possible for UDP packets to be
discarded, and
> > the way we deal with this in xAP is to accept that this
can
> > happen to xAP messages, and layer application level
> > acknowledgements where knowing that a message has been
> > received is critically important - whether explicitly or
> > implicitly through status messages. There are various
schemes
> > that could be adopted to allow a receiver to detect lost
> > messages (e.g. sequence numbering), but they quickly
become
> > quite onerous, and assume that the originator keeps a
copy of
> > the original message or is able to reconstruct it - which
is
> > non-trivial for the many xAP nodes.
> >
> > Perhaps you can share more of the specific details of the
> > device(s) in question (manufacturer, docs, o/s etc), and
their
> > specific behaviour, which seems a bit anomalous?
> >
> > Patrick
> >
> > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>>
> >
> > One of the issues seems to be that there is conflicting
views
> > as to the
> > length of a a UDP data packet payload. Some people cite
500
> > or 512
> > characters and some 1500. Regardless in some low
> > memory/performance
> > devices it is being reported, that even with UDP,
packets are
> > being
> > received from the buffer either truncated or appended
back to
> > back.
> > The latter I assume is due to the speed of the device in
> > servicing the
> > buffer.
> >
> > We have an opportunity to protect against this with v1.3
and
> > the double
> > '0A' seems the most compatible. I would be loathe to
support
> > anything
> > that wasn't backward compatible.
> >
> > K
> >
> >
> > Patrick Lidstone wrote:
> > >
> > >
> > > I will dig it out - it included an optional checksum
I
> > think, and IIRC
> > > was framed by stx and etx (a kind of pseudo industry
> > standard). I
> > > certainly used it with the PIC serial stuff and the
> > xAP-serial bridge.
> > > Re.: long message truncation and concatenation: If
we need
> > to support
> > > messages that are larger than one UDP packet, then
there are
> > > additional complexities and the proposed scheme
won't work
> > as intended
> > > as the ordering of UDP messages is not guaranteed.
I'm happy
> > to help
> > > refine a spec to support these capabilities, but it
is
> > moving away
> > > from the basic ethos of xAP somewhat, as devices
would have
> > to be able
> > > to buffer received messages, and that raises the bar
> > considerably.
> > > Perhaps there is scope for co-existence of a long
and
> > standard message
> > > protocol though?
> > >
> > > Patrick
> > >
> > > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>
> >
> > > <mailto:yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>>>
> >
> > >
> > > ... Oh ... where is that in the spec ? it might
be all
> > we need.
> > >
> > > This is also tied in with some aspects of
long
> > message truncation
> > > and concatenation of messages received in UDP
receive
> > buffers
> > > though...
> > >
> > > K
> > >
> > > Patrick Lidstone wrote:
> > > >
> > > >
> > > > The original xap spec provides extensions
for framing
> > a message over
> > > > async serial which also delimit the start
of the
> > message - you don't
> > > > need this 'hack' if you follow the original
spec for
> > non-UDP
> > > transports.
> > > >
> > > > Patrick
> > > >
> > > > 2009/9/13 Kevin Hawkins
<yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>
> > > <mailto:yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>>
> >
> > > > <mailto:yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>
> >
> > > <mailto:yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>>>>
> > > >
> > > > We have been asked on several
occasions how to
> > detect the
> > > end of a
> > > > xAP messager as there is no unique EOM
character.
> > Typically
> > > in any
> > > > reasonable sized packet structured
transport eg
> > UDP then the
> > > packet
> > > > provides such an indication but on
systems with
> > small packets or
> > > > non eg
> > > > asynchronous serial this is not
useable.
> > > >
> > > > In discussing this with the
specification team
> > we must
> > > consider
> > > > backwards compatability and so we do
not wish to
> > alter the
> > > > specification
> > > > to include a unique EOM character.
What we do
> propose
> > > however is that
> > > > xAP v1.3 will specify that all messages
should end
> > with two
> > > > consecutive
> > > > chr(10)'s immediately after the closing
'}'
> > > >
> > > > ie ..... 7D 0A 0A
> > > >
> > > > Some apps, even v1.2 ones, already do
this. We
> > don't
> > > believe this
> > > > will cause any backwards compatibility
issues and
> > it will
> > > always be
> > > > unique within a xAP message.
> > > >
> > > > So, in developing xAP v1.3 apps could
you
> > therefore append
> > > two 0A's
> > > > at the end of your messages , and of
course handle
> > incoming
> > > messages
> > > > containing such.
> > > >
> > > > K
> > > >
> > > >
> > > > ------------------------------------
> > > >
> > > > Yahoo! Groups Links
> > > >
> > > >
> > > >
> > mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> > > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>>>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > ------------------------------------
> > >
> > > Yahoo! Groups Links
> > >
> > >
> > > mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> > mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>
--00163600ce79dd64220473c36751
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
<head>
<style type=3D"text/css">
<!--
/* start of attachment style */
.ygrp-photo-title{
clear: both;
font-size: smaller;
height: 15px;
overflow: hidden;
text-align: center;
width: 75px;
}
div.ygrp-photo{
background-position: center;
background-repeat: no-repeat;
background-color: white;
border: 1px solid black;
height: 62px;
width: 62px;
}
div.photo-title=20
a,
div.photo-title a:active,
div.photo-title a:hover,
div.photo-title a:visited {
text-decoration: none;=20
}
div.attach-table div.attach-row {
clear: both;
}
div.attach-table div.attach-row div {
float: left;
/* margin: 2px;*/
}
p {
clear: both;
padding: 15px 0 3px 0;
overflow: hidden;
}
div.ygrp-file {
width: 30px;
valign: middle;
}
div.attach-table div.attach-row div div a {
text-decoration: none;
}
div.attach-table div.attach-row div div span {
font-weight: normal;
}
div.ygrp-file-title {
font-weight: bold;
}
/* end of attachment style */
-->
</style>
</head>
<html><head>
<style type=3D"text/css">
<!--
#ygrp-mkp{
border: 1px solid #d8d8d8;
font-family: Arial;
margin: 14px 0px;
padding: 0px 14px;
}
#ygrp-mkp hr{
border: 1px solid #d8d8d8;
}
#ygrp-mkp #hd{
color: #628c2a;
font-size: 85%;
font-weight: bold;
line-height: 122%;
margin: 10px 0px;
}
#ygrp-mkp #ads{
margin-bottom: 10px;
}
#ygrp-mkp .ad{
padding: 0 0;
}
#ygrp-mkp .ad a{
color: #0000ff;
text-decoration: none;
}
-->
</style>
</head>
<body>
<!-- **begin egp html banner** -->
<br><br>
<!-- **end egp html banner** -->
How about keeping it in-band (as xAP messages) but using it as the
foundati=
on for a generic configuration schema, which is looooong overdue! Would
tha=
t work? There may be some benefit from other end points being aware of
filt=
ering changes etc.? Authentication would have to be a closed dialogue, but
=
could still be xAP, between just the hub and client.<br>
<br>(A tcp hub could just pass config schema messages and heartbeats
by def=
ault on start up perhaps?)<br><br>P.<br><br><div
class=3D"gmail_quote">2009=
/9/17 Kevin Hawkins <span dir=3D"ltr"><<a
href=3D"mailto:yahoogroupskh@g=
ooglemail.com">yahoogroupskh@xxxxxxx</a>></span><br>
<blockquote class=3D"gmail_quote" style=3D"border-left:
1px solid rgb(204, =
204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">If we
wish to all=
ow some inband mechansim for a TCP client to configure<br>
it's link eg for logon/authentication, selecting a preset
profile<br>
governing which traffic is passed or to be able to create dynamic<br>
filters then we need some way to extract link control traffic.
Perhaps<br>
we could do this via a specific xAP schema targeted at the
hub/router<br>
(?) . =A0The iServer currently uses a different
<stx><etx> comb=
ination for<br>
control and xAP traffic, rather than any escape sequence or auto
detect<br>
of non xAP content. This allows mid session changes. The specifics
could<br=
>
easily be changed as AFAIK there are only two people using the
dynamic<br>
filtering feature. =A0It would be tody to allow the<br>
xRouter/xServer/iServer to be able to connect bidirectionally to
a<br>
standard release hub.<br>
<br>
=A0 K<br>
<div class=3D"im"><br>
=A0Patrick Lidstone wrote:<br>
><br>
><br>
> Cool!<br>
><br>
> I'm not sure the transport wrapper over TCP
'breaks everything=
' - I've<br>
> used it for TCP routing of xAP over the internet framed using
the<br>
> transport wrapper for several years now (I even demoed it as a
means<b=
r>
> of remotely accessing my home intranet via xAP at the inaugural
HA<br>
> weekend meet many moons ago), but perhaps I'm missing
something fr=
om<br>
> your requirements? All the tcp hub has to do is strip off the
start<br=
>
> and end delimiter before rebroadcast, and likewise frame any
received<=
br>
> udp messages with an <stx>, <etx> if
operating bi-directio=
nally.<br>
> Admittedly it's an additional (minimal) cost that
isn't presen=
t with<br>
> the double-0A scheme, but the benefits of being able to detect
the<br>
> start of frame, and the benefits of being able to apply this
approach<=
br>
> to any async transport, not just tcp/ip, outweigh that in my mind.
And=
<br>
> it is in the spec already ;-)<br>
><br>
> Personally, I don't see a need for a delimiter at the end
of a mes=
sage<br>
> in UDP xAP. I'm sure I'm going to be outvoted, but
look at all=
the<br>
> other UDP protocols out there - it's not common practice
because i=
t's<br>
> not technically necessary, and as such it's just adding
unnecessar=
y<br>
> cruft to xAP!<br>
><br>
> Cheers<br>
> Patrick<br>
><br>
><br>
> 2009/9/17 Edward Pearson <<a href=3D"mailto:edward.mailgroup@blueyo=
nder.co.uk">edward.mailgroup@xxxxxxx</a><br>
</div>> <mailto:<a href=3D"mailto:edward.mailgroup@xxxxxxx";>=
edward.mailgroup@xxxxxxx</a>>><br>
<div><div></div><div
class=3D"h5">><br>
><br>
><br>
> =A0 =A0 You=92re right, to get a connection in good time that=92s
not<=
br>
> =A0 =A0 statically configured it=92d need a request/offer message
pair=
.<br>
><br>
><br>
><br>
> =A0 =A0 The experimental hub I have here is designed to address
one<br=
>
> =A0 =A0 specific problem only =96 unreliability over
WiFi.<br>
><br>
> =A0 =A0 In runs as a master slave arrangement. In master mode it
accep=
ts<br>
> =A0 =A0 incoming tcp connections on port 3639. In slave mode
(deployed=
on<br>
> =A0 =A0 the WiFi machine) it=92s configured (statically) with the
IP a=
ddress<br>
> =A0 =A0 or hostname of the master hub to which it connects over
tcp. T=
he<br>
> =A0 =A0 tcp connection is used one-way only, from master to slave.
The=
<br>
> =A0 =A0 master forwards all xAP messages it receives (on udp) to
all i=
ts<br>
> =A0 =A0 local udp clients (as usual) and to any connected tcp
clients.=
The<br>
> =A0 =A0 slave forwards all messages received on its tcp connection
to =
its<br>
> =A0 =A0 local clients on udp. Works a treat. Now I can lie on the
sofa=
<br>
> =A0 =A0 debugging Viewer v4 (it=92ll be out really, really soon
now I<=
br>
> =A0 =A0 promise) on my WiFi laptop without wondering where 15% of
the<=
br>
> =A0 =A0 messages have gone.<br>
><br>
><br>
><br>
> =A0 =A0 I mention all this mainly (and at risk of getting back to
Kevi=
n=92s<br>
> =A0 =A0 original point of this thread which I=92m feeling guilty
of<br=
>
> =A0 =A0 hijacking horribly) because I needed to address the
problem of=
the<br>
> =A0 =A0 segregation =A0of messages in a stream to get this to
work. So=
I<br>
> =A0 =A0 experimented with several schemes =A0getting a good feel
for t=
he<br>
> =A0 =A0 practical pro=92s and cons of each. Initially I went for
the S=
TX/ETX<br>
> =A0 =A0 method from the transport wrapper since it=92s what=92s in
the=
<br>
> =A0 =A0 existing spec (but opting out of the CRC). Later I used
the do=
uble<br>
> =A0 =A0 LF termination. The transport wrapper is somewhat easier
to wo=
rk<br>
> =A0 =A0 with. For instance, it=92s easier to re-sync to the a
message =
start<br>
> =A0 =A0 with transport wrapper; just wait for the next STX (a
single<b=
r>
> =A0 =A0 byte), vs. needing match all of
=93xap-[headerhbeat]<lf>=
;=94. Message<br>
> =A0 =A0 ends are similar: ETX (1 byte) vs
<lf><lf> (2 byte=
s). =A0But all<br>
> =A0 =A0 this is irrelevant in the udp world. Introducing STX/ETX
here<=
br>
> =A0 =A0 would break everything while double <lf> is
nicely backw=
ards<br>
> =A0 =A0 compatible.<br>
><br>
><br>
><br>
> =A0 =A0 (and finally, my point relevant to the thread
start)<br>
><br>
><br>
><br>
> =A0 =A0 I don=92t think double <lf> is the right
solution for Ke=
vin=92s<br>
> =A0 =A0 developers=92 wireless app problem; it=92d help them out
somew=
hat but<br>
> =A0 =A0 I think it more than likely that it=92s =91papering
over=92 a =
bug in or<br>
> =A0 =A0 around their network stack. But there is a valid need
lurking =
in<br>
> =A0 =A0 here; without message-level delimiting in the original
spec,<b=
r>
> =A0 =A0 implementers of xAP commonly run up against the problem of
kno=
wing<br>
> =A0 =A0 where the end of a message is while parsing messages. The
doub=
le<br>
> =A0 =A0 <lf> definitely helps here. So given that
it=92s easy to=
add, helps<br>
> =A0 =A0 in some folks, eases adoption and causes no
backwards<br>
> =A0 =A0 compatibility issues, I=92m happy that it should go into
the v=
1.3 spec.<br>
><br>
><br>
><br>
> =A0 =A0 *From:* <a href=3D"mailto:xAP_developer@xxxxxxx";>xAP_d=
eveloper@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer@xxxxxxx";>xA=
P_developer@xxxxxxx</a>><br>
> =A0 =A0 [mailto:<a href=3D"mailto:xAP_developer@xxxxxxx";>xAP_d=
eveloper@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer@xxxxxxx";>xA=
P_developer@xxxxxxx</a>>] *On Behalf Of *Patrick<br>
> =A0 =A0 Lidstone<br>
> =A0 =A0 *Sent:* 16 September 2009 12:00<br>
><br>
> =A0 =A0 *To:* <a href=3D"mailto:xAP_developer@xxxxxxx";>xAP_dev=
eloper@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer@xxxxxxx";>xA=
P_developer@xxxxxxx</a>><br>
</div></div>> =A0 =A0 *Subject:* Re: [xAP_developer] xAP
EOM identifier =
in xAP v1.3<br>
<div class=3D"im">><br>
><br>
><br>
><br>
><br>
> =A0 =A0 I agree that checksums are not appropriate to TCP
connections,=
the<br>
> =A0 =A0 transport wrapper already designates them as
optional.<br>
><br>
> =A0 =A0 I don't think hubs currently send heartbeats? If
they do, =
it's not<br>
> =A0 =A0 explicit in the spec. (Mine doesn't). We also need
a mecha=
nism to<br>
> =A0 =A0 differentiate between multiple hubs (since most people
will<br=
>
> =A0 =A0 generally run at least one hub per vm).<br>
><br>
> =A0 =A0 Is a TCP hub going to relay all traffic to all TCP
endpoints? =
That<br>
> =A0 =A0 won't scale very well... but it's also not
clear how t=
he filtering<br>
> =A0 =A0 etc. will work. Administering filters manually works fine
for =
the<br>
> =A0 =A0 odd serial segment, but it's not going to be
viable for a =
large<br>
> =A0 =A0 number of endpoints.<br>
><br>
> =A0 =A0 Heartbeats are broadcast every minute or so typically, so
that=
's<br>
> =A0 =A0 going to result in start up times of up to two minutes for
a t=
cp<br>
> =A0 =A0 end point. Is that acceptable? If not, do we reduce the
new tc=
p<br>
> =A0 =A0 hub heartbeat interval? Or is there a mechanism for an
endpoin=
t to<br>
> =A0 =A0 poke a hub and elicit a heartbeat?<br>
><br>
> =A0 =A0 And lastly, is there merit in some kind of referrer system
to<=
br>
> =A0 =A0 allow for load-balancing and automated failover of tcp
connect=
ions<br>
> =A0 =A0 across more than one hub? If so, how would this
work?<br>
><br>
> =A0 =A0 Patrick<br>
><br>
> =A0 =A0 2009/9/15 Edward Pearson <<a href=3D"mailto:edward.mailgrou=
p@xxxxxxx">edward.mailgroup@xxxxxxx</a><br>
</div>> =A0 =A0 <mailto:<a href=3D"mailto:edward.mailgroup@blueyonder=
.co.uk">edward.mailgroup@xxxxxxx</a>>><br>
<div><div></div><div
class=3D"h5">><br>
><br>
><br>
> =A0 =A0 I don=92t mean that packet fragmentation or re-ordering
are my=
ths =96<br>
> =A0 =A0 just that when dealing with xAP at the IP stack (socket)
inter=
face<br>
> =A0 =A0 you are dealing with datagrams not data packets so you
don=92t=
need<br>
> =A0 =A0 to worry about those packet aspects. There was a design
decisi=
on<br>
> =A0 =A0 to limit xAP datagrams to 1500 bytes to improve the
likely<br>
> =A0 =A0 coverage of correctly sent datagrams in a world of IP
stacks o=
f<br>
> =A0 =A0 varying quality. It=92s no more important than that.
Readers o=
f the<br>
> =A0 =A0 spec seem to attach more importance to it than it needs
(there=
=92s<br>
> =A0 =A0 the myth aspect). =A0To program xAP you don=92t need to
worry =
about<br>
> =A0 =A0 fragmentation and reordering =96 just keep your datagrams
1500=
bytes<br>
> =A0 =A0 or less and let the stack do its thing.<br>
><br>
><br>
><br>
> =A0 =A0 Sometimes an app goes a bit wild (xAP news has
occasionally be=
en a<br>
> =A0 =A0 useful test source) and big xAP messages are generated =96
and=
they<br>
> =A0 =A0 get delivered too! It=92s normally the input buffer of the
rec=
eiving<br>
> =A0 =A0 app (eg hub) that breaks first.<br>
><br>
><br>
><br>
> =A0 =A0 On the tcp framing, I=92d suggest that implementing the
CRC pa=
rt<br>
> =A0 =A0 (irrelevant on a reliable stream) is a waste of most
people=92=
s<br>
> =A0 =A0 time; by far more use will be made of tcp than any kind
of<br>
> =A0 =A0 serial/485/etc networks so they=92d be sharing development
of =
an<br>
> =A0 =A0 implementation with nobody.<br>
><br>
><br>
><br>
> =A0 =A0 Discovery can be done simply with the existing hub
heartbeat<b=
r>
> =A0 =A0 message. Just need to agree on an extra block that
advertises =
the<br>
> =A0 =A0 port number of the tcp service =96 which I assume, by
default =
would<br>
> =A0 =A0 be 3639.<br>
><br>
><br>
><br>
> =A0 =A0 Having read the 802.11 spec, I now understand why
broadcast ud=
p<br>
> =A0 =A0 from an AP to a NIC is so un-reliable. And ad-hoc mode is
a re=
al<br>
> =A0 =A0 disaster!<br>
><br>
><br>
><br>
> =A0 =A0 *From:* <a href=3D"mailto:xAP_developer@xxxxxxx";>xAP_d=
eveloper@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer@xxxxxxx";>xA=
P_developer@xxxxxxx</a>><br>
</div></div><div class=3D"im">> =A0 =A0
[mailto:<a href=3D"mailto:xAP_de=
veloper@xxxxxxx">xAP_developer@xxxxxxx</a><br>
</div><div class=3D"im">> =A0 =A0
<mailto:<a href=3D"mailto:xAP_devel=
oper@xxxxxxx">xAP_developer@xxxxxxx</a>>] *On Behalf
Of =
*Patrick<br>
> =A0 =A0 Lidstone<br>
> =A0 =A0 *Sent:* 15 September 2009 14:15<br>
><br>
><br>
> =A0 =A0 *To:* <a href=3D"mailto:xAP_developer@xxxxxxx";>xAP_dev=
eloper@xxxxxxx</a><br>
</div><div class=3D"im">> =A0 =A0
<mailto:<a href=3D"mailto:xAP_devel=
oper@xxxxxxx">xAP_developer@xxxxxxx</a>><br>
</div>> =A0 =A0 *Subject:* Re: [xAP_developer] xAP EOM
identifier in xAP=
v1.3<br>
<div class=3D"im">><br>
><br>
><br>
><br>
><br>
> =A0 =A0 Hi Edward,<br>
> =A0 =A0 Please see in-line responses...<br>
><br>
> =A0 =A0 2009/9/15 Edward Pearson <<a href=3D"mailto:edward.mailgrou=
p@xxxxxxx">edward.mailgroup@xxxxxxx</a><br>
</div>> =A0 =A0 <mailto:<a href=3D"mailto:edward.mailgroup@blueyonder=
.co.uk">edward.mailgroup@xxxxxxx</a>>><br>
<div><div></div><div
class=3D"h5">><br>
><br>
><br>
> =A0 =A0 The double 0x0a terminator works for me, it=92s simple to
impl=
ement<br>
> =A0 =A0 =96 already done for my stuff.<br>
><br>
><br>
><br>
> =A0 =A0 ...but it isn't robust unless the message is sent
as a sin=
gle<br>
> =A0 =A0 datagram, and if it is sent as a single datagram, the os
shoul=
d<br>
> =A0 =A0 deliver it as such -- and if it doesn't, adding an
eom mar=
ker<br>
> =A0 =A0 isn't going to help.<br>
><br>
> =A0 =A0 =A0 =A0 For reliable streams, such as TCP, I generally
frame t=
he<br>
> =A0 =A0 =A0 =A0 messages with STX and ETX.<br>
><br>
><br>
><br>
> =A0 =A0 =A0 =A0 I thought all this business about 1500 bytes was
somew=
hat<br>
> =A0 =A0 =A0 =A0 urban myth (at least as it applies to xAP). The
data p=
acket<br>
> =A0 =A0 =A0 =A0 size is not what=92s important; it=92s how the
particu=
lar IP stack<br>
> =A0 =A0 =A0 =A0 implementation deals with
>datagrams< that=92s t=
he key.<br>
><br>
><br>
> =A0 =A0 I don't understand what you are syaing here? The
data pack=
et size<br>
> =A0 =A0 is inextricably linked to the IP stack. What aspect is it
that=
you<br>
> =A0 =A0 consider to be an urban myth?<br>
><br>
><br>
> =A0 =A0 =A0 =A0 I only have experience of the two most recent
Windows =
stacks<br>
> =A0 =A0 =A0 =A0 (XP and Vista) which I agree are likely more
capable t=
han old<br>
> =A0 =A0 =A0 =A0 embedded stacks. My experiments with Wireshark
show th=
at those<br>
> =A0 =A0 =A0 =A0 stacks will happily deal with fragmentation
and<br>
> =A0 =A0 =A0 =A0 defragmentation. 64KB in a single datagram? No
problem=
if your<br>
> =A0 =A0 =A0 =A0 receive buffer is long enough (even if you force a
sma=
ll MTU).<br>
><br>
> =A0 =A0 Yes, agreed, OSes perform fragmentation for you. The
individua=
l<br>
> =A0 =A0 fragments have a maximum size as determined by the MTU.
For<br=
>
> =A0 =A0 pragmatic, practical reasons, xAP needs to define a
maximum<br=
>
> =A0 =A0 overall message size, and for convenience's sake
that was =
set as<br>
> =A0 =A0 equal to the 'standard' MTU size. Devices
which use a =
smaller MTU<br>
> =A0 =A0 /should /fragment and reassemble seamlessly provided that
the<=
br>
> =A0 =A0 correct socket options have been set to define the maximum
UDP=
<br>
> =A0 =A0 packet size. By electing to use a single MTU's
worth of da=
ta, we<br>
> =A0 =A0 we avoid the overhead associated with fragmentation and
reasse=
mbly<br>
> =A0 =A0 (principally memory buffering) which is a good thing.
When<br>
> =A0 =A0 reassembled, the os should deliver a datagram as a
complete en=
tity<br>
> =A0 =A0 in one go (irrespective of the mtu size, assuming that
the<br>
> =A0 =A0 datagram falls within the maximum datagram size defined
for th=
e<br>
> =A0 =A0 stack). If the sender doesn't send a message as a
single d=
atagram,<br>
> =A0 =A0 then the whole thing falls apart because effectively you
are t=
hen<br>
> =A0 =A0 doing fragmentation and reassembly at the application
layer, a=
nd<br>
> =A0 =A0 that won't work because the ordering across
datagrams is n=
ot<br>
> =A0 =A0 guaranteed and individual datagrams may get
lost.<br>
><br>
><br>
> =A0 =A0 =A0 =A0 If anything goes wrong (eg, missing fragment) the
whol=
e<br>
> =A0 =A0 =A0 =A0 datagram is dropped. I can=92t see how packet
re-order=
ing can<br>
> =A0 =A0 =A0 =A0 happen on a single LAN for broadcast UDP =96 there
are=
no<br>
> =A0 =A0 =A0 =A0 multiple paths and no retry mechanism. Certainly
never=
<br>
> =A0 =A0 =A0 =A0 observed it =96 I assume the stack would again
just dr=
op the<br>
> =A0 =A0 =A0 =A0 entire datagram.<br>
><br>
><br>
> =A0 =A0 Re-ordering of datagrams can occur for multiple reasons on
a<b=
r>
> =A0 =A0 single lan. A sender may send individual UDP packets in
any or=
der<br>
> =A0 =A0 it chooses. This commonly occurs with fragmented
packets<br>
> =A0 =A0 originating from a multi-threaded sender, which can be
interle=
aved<br>
> =A0 =A0 with smaller, non-fragmented datagrams as required
(optimising=
<br>
> =A0 =A0 transient memory use, as soon as they are sent, the buffer
can=
be<br>
> =A0 =A0 released). A switch is not obliged to retransmit packets
in th=
e<br>
> =A0 =A0 order in which they are received. And most fundamentally,
the<=
br>
> =A0 =A0 specs do not require UDP packets to be ordered so, even if
you=
<br>
> =A0 =A0 don't observe it, it /can/ happen, and sooner or
later<br>
> =A0 =A0 interoperability issues will arise if the working
assumption i=
s<br>
> =A0 =A0 made that they always arrive in order.<br>
><br>
><br>
><br>
> =A0 =A0 =A0 =A0 A bigger issue for me, and the reason for me
experimen=
ting a<br>
> =A0 =A0 =A0 =A0 few times with TCP stream serving from hubs, is
datagr=
am loss<br>
> =A0 =A0 =A0 =A0 over WiFi networks. This is greatly accentuated
for UD=
P when<br>
> =A0 =A0 =A0 =A0 you use broadcast (as we do) from wired to
wireless (f=
ine the<br>
> =A0 =A0 =A0 =A0 other way as it=92s not really a broadcast packet
till=
it gets<br>
> =A0 =A0 =A0 =A0 to the AP) as the access point and NIC can not
longer =
apply<br>
> =A0 =A0 =A0 =A0 their ack/nak procedure at the transport level. I
comm=
only see<br>
> =A0 =A0 =A0 =A0 xAP datagram loss rates from wired to wireless
connect=
ions of<br>
> =A0 =A0 =A0 =A0 20%. So I=92d like us to agree on a standard
transport=
wrapper<br>
> =A0 =A0 =A0 =A0 for TCP streams which a lot of platforms would
find us=
eful.<br>
><br>
><br>
><br>
> =A0 =A0 I'd suggest using the same framing as the
"transport =
wrapper", as<br>
> =A0 =A0 this then allows for code to be shared across transports.
If x=
AP<br>
> =A0 =A0 was extended to support TCP, then that should also include
a<b=
r>
> =A0 =A0 formal discovery mechanism by which the IP
address/characteris=
tics<br>
> =A0 =A0 of the hub can be discovered (over UDP xAP?)<br>
><br>
> =A0 =A0 Patrick<br>
><br>
><br>
> =A0 =A0 =A0 =A0 *From:* <a href=3D"mailto:xAP_developer@xxxxxxx=
m">xAP_developer@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer@yahoogroups=
.com">xAP_developer@xxxxxxx</a>><br>
</div></div><div class=3D"im">> =A0 =A0
=A0 =A0 [mailto:<a href=3D"mailt=
o:xAP_developer@xxxxxxx">xAP_developer@xxxxxxx</a><br>
</div><div class=3D"im">> =A0 =A0 =A0 =A0
<mailto:<a href=3D"mailto:x=
AP_developer@xxxxxxx">xAP_developer@xxxxxxx</a>>] *On
Be=
half Of *Patrick<br>
> =A0 =A0 =A0 =A0 Lidstone<br>
> =A0 =A0 =A0 =A0 *Sent:* 14 September 2009 14:19<br>
> =A0 =A0 =A0 =A0 *To:* <a href=3D"mailto:xAP_developer@xxxxxxx"=
>xAP_developer@xxxxxxx</a><br>
</div><div class=3D"im">> =A0 =A0 =A0 =A0
<mailto:<a href=3D"mailto:x=
AP_developer@xxxxxxx">xAP_developer@xxxxxxx</a>><br>
</div>> =A0 =A0 =A0 =A0 *Subject:* Re: [xAP_developer] xAP EOM
identifie=
r in xAP v1.3<br>
<div><div></div><div
class=3D"h5">><br>
><br>
><br>
><br>
><br>
> =A0 =A0 =A0 =A0 So the xAP delimiters for serial are defined in
the 1.=
2 xAP<br>
> =A0 =A0 =A0 =A0 spec here:<br>
> =A0 =A0 =A0 =A0 <a href=3D"http://www.xapautomation.org/index.php?titl=
e=3DProtocol_definition#Transport_Wrapper"
target=3D"_blank">http://www.xap=
automation.org/index.php?title=3DProtocol_definition#Transport_Wrapper</a><=
br>
><br>
> =A0 =A0 =A0 =A0 To the best of my knowledge, the MTU size for an
ether=
net<br>
> =A0 =A0 =A0 =A0 frame is 1518 bytes, which leads to a UDP packet
MTU o=
f 1500<br>
> =A0 =A0 =A0 =A0 bytes and this is the size that is adopted by the
majo=
rity of<br>
> =A0 =A0 =A0 =A0 operating systems. Internet routers (i.e. ISPs)
someti=
mes use<br>
> =A0 =A0 =A0 =A0 an MTU of 576 bytes, but this wouldn't be
relevant=
to xAP<br>
> =A0 =A0 =A0 =A0 since the traffic doesn't pass over the
wider net,=
or if it<br>
> =A0 =A0 =A0 =A0 does, it's generally gatewayed via a
TCP/IP connec=
tion.<br>
><br>
> =A0 =A0 =A0 =A0 If a device is receiving fragmented UDP packets, I
thi=
nk the<br>
> =A0 =A0 =A0 =A0 same issues arise as those related to extending
the xA=
P<br>
> =A0 =A0 =A0 =A0 message size beyond 1500 bytes - what happens if a
fra=
gment<br>
> =A0 =A0 =A0 =A0 gets discarded.<br>
><br>
> =A0 =A0 =A0 =A0 If you take the scenario:<br>
><br>
> =A0 =A0 =A0 =A0 Part 1(a), Part 1(b), Part 2(a), Part 2(b), Part
3(a),=
Part 3(b)<br>
><br>
> =A0 =A0 =A0 =A0 --- first there is no guarantee that the parts
will be=
<br>
> =A0 =A0 =A0 =A0 delivered in order, and second, if part 1(b) was
dropp=
ed, and<br>
> =A0 =A0 =A0 =A0 you were blindly assembling messages based on the
prop=
osed<br>
> =A0 =A0 =A0 =A0 double-0a terminator, you'd end up with a
message =
comprising<br>
> =A0 =A0 =A0 =A0 part 1(a), 2(a) and 2(b) which is not only
obviously c=
orrupt,<br>
> =A0 =A0 =A0 =A0 but also possibly larger than the maximum xAP
message =
size,<br>
> =A0 =A0 =A0 =A0 blowing away your buffers.<br>
><br>
> =A0 =A0 =A0 =A0 I think the solution is to probably parse incoming
mes=
sages on<br>
> =A0 =A0 =A0 =A0 the fly, byte-by-byte. You can then at least reset
you=
r state<br>
> =A0 =A0 =A0 =A0 when you encounter the xap-header, and if you
count op=
en and<br>
> =A0 =A0 =A0 =A0 close curly braces, you can tell when you have an
appa=
rently<br>
> =A0 =A0 =A0 =A0 complete message. This won't solve the
issue of UD=
P fragments<br>
> =A0 =A0 =A0 =A0 being potentially received out of order, but so
long a=
s we are<br>
> =A0 =A0 =A0 =A0 dealing with a single LAN, and fragmentation
occurs at=
the<br>
> =A0 =A0 =A0 =A0 receiver, we will be ok I think.<br>
><br>
> =A0 =A0 =A0 =A0 It is absolutely possible for UDP packets to be
discar=
ded, and<br>
> =A0 =A0 =A0 =A0 the way we deal with this in xAP is to accept that
thi=
s can<br>
> =A0 =A0 =A0 =A0 happen to xAP messages, and layer application
level<br=
>
> =A0 =A0 =A0 =A0 acknowledgements where knowing that a message has
been=
<br>
> =A0 =A0 =A0 =A0 received is critically important - whether
explicitly =
or<br>
> =A0 =A0 =A0 =A0 implicitly through status messages. There are
various =
schemes<br>
> =A0 =A0 =A0 =A0 that could be adopted to allow a receiver to
detect lo=
st<br>
> =A0 =A0 =A0 =A0 messages (e.g. sequence numbering), but they
quickly b=
ecome<br>
> =A0 =A0 =A0 =A0 quite onerous, and assume that the originator
keeps a =
copy of<br>
> =A0 =A0 =A0 =A0 the original message or is able to reconstruct it
- wh=
ich is<br>
> =A0 =A0 =A0 =A0 non-trivial for the many xAP nodes.<br>
><br>
> =A0 =A0 =A0 =A0 Perhaps you can share more of the specific details
of =
the<br>
> =A0 =A0 =A0 =A0 device(s) in question (manufacturer, docs, o/s
etc), a=
nd their<br>
> =A0 =A0 =A0 =A0 specific behaviour, which seems a bit
anomalous?<br>
><br>
> =A0 =A0 =A0 =A0 Patrick<br>
><br>
> =A0 =A0 =A0 =A0 2009/9/13 Kevin Hawkins <<a
href=3D"mailto:yahoogro=
upskh@xxxxxxx">yahoogroupskh@xxxxxxx</a><br>
</div></div>> =A0 =A0 =A0 =A0 <mailto:<a
href=3D"mailto:yahoogroupskh=
@googlemail.com">yahoogroupskh@xxxxxxx</a>>><br>
<div><div></div><div
class=3D"h5">><br>
> =A0 =A0 =A0 =A0 One of the issues seems to be that there is
conflictin=
g views<br>
> =A0 =A0 =A0 =A0 as to the<br>
> =A0 =A0 =A0 =A0 length of a a UDP data packet payload. =A0Some
people =
cite 500<br>
> =A0 =A0 =A0 =A0 or 512<br>
> =A0 =A0 =A0 =A0 characters and some 1500. =A0Regardless in some
low<br=
>
> =A0 =A0 =A0 =A0 memory/performance<br>
> =A0 =A0 =A0 =A0 devices it is being reported, that even with UDP,
=A0p=
ackets are<br>
> =A0 =A0 =A0 =A0 being<br>
> =A0 =A0 =A0 =A0 received from the buffer either truncated or
appended =
back to<br>
> =A0 =A0 =A0 =A0 back.<br>
> =A0 =A0 =A0 =A0 The latter I assume is due to the speed of the
device =
in<br>
> =A0 =A0 =A0 =A0 servicing the<br>
> =A0 =A0 =A0 =A0 buffer.<br>
><br>
> =A0 =A0 =A0 =A0 We have an opportunity to protect against this
with v1=
.3 and<br>
> =A0 =A0 =A0 =A0 the double<br>
> =A0 =A0 =A0 =A0 '0A' seems the most compatible.
=A0 I would be=
loathe to support<br>
> =A0 =A0 =A0 =A0 anything<br>
> =A0 =A0 =A0 =A0 that wasn't backward compatible.<br>
><br>
> =A0 =A0 =A0 =A0 =A0 K<br>
><br>
><br>
> =A0 =A0 =A0 =A0 Patrick Lidstone wrote:<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > I will dig it out - it included an
optional check=
sum I<br>
> =A0 =A0 =A0 =A0 think, and IIRC<br>
> =A0 =A0 =A0 =A0 > was framed by stx and etx (a kind of
pseudo indus=
try<br>
> =A0 =A0 =A0 =A0 standard). I<br>
> =A0 =A0 =A0 =A0 > certainly used it with the PIC serial
stuff and t=
he<br>
> =A0 =A0 =A0 =A0 xAP-serial bridge.<br>
> =A0 =A0 =A0 =A0 > Re.: long message truncation and
concatenation: I=
f we need<br>
> =A0 =A0 =A0 =A0 to support<br>
> =A0 =A0 =A0 =A0 > messages that are larger than one UDP
packet, the=
n there are<br>
> =A0 =A0 =A0 =A0 > additional complexities and the proposed
scheme w=
on't work<br>
> =A0 =A0 =A0 =A0 as intended<br>
> =A0 =A0 =A0 =A0 > as the ordering of UDP messages is not
guaranteed=
. I'm happy<br>
> =A0 =A0 =A0 =A0 to help<br>
> =A0 =A0 =A0 =A0 > refine a spec to support these
capabilities, but =
it is<br>
> =A0 =A0 =A0 =A0 moving away<br>
> =A0 =A0 =A0 =A0 > from the basic ethos of xAP somewhat, as
devices =
would have<br>
> =A0 =A0 =A0 =A0 to be able<br>
> =A0 =A0 =A0 =A0 > to buffer received messages, and that
raises the =
bar<br>
> =A0 =A0 =A0 =A0 considerably.<br>
> =A0 =A0 =A0 =A0 > Perhaps there is scope for co-existence
of a long=
and<br>
> =A0 =A0 =A0 =A0 standard message<br>
> =A0 =A0 =A0 =A0 > protocol though?<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > Patrick<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > 2009/9/13 Kevin Hawkins <<a
href=3D"mailto:yah=
oogroupskh@xxxxxxx">yahoogroupskh@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@googlemail.=
com">yahoogroupskh@xxxxxxx</a>><br>
><br>
</div></div><div><div></div><div
class=3D"h5">> =A0 =A0 =A0 =A0 > <=
;mailto:<a href=3D"mailto:yahoogroupskh@xxxxxxx";>yahoogroupskh@googl=
email.com</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@googlemail.=
com">yahoogroupskh@xxxxxxx</a>>>><br>
><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ... Oh ... where is that in the
spec ? =
=A0it might be all<br>
> =A0 =A0 =A0 =A0 we need.<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 =A0 =A0This is also tied in with
some asp=
ects of long<br>
> =A0 =A0 =A0 =A0 message truncation<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 and concatenation of messages
received in=
UDP receive<br>
> =A0 =A0 =A0 =A0 buffers<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 though...<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 =A0K<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 Patrick Lidstone wrote:<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > The original xap spec
provides exten=
sions for framing<br>
> =A0 =A0 =A0 =A0 a message over<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > async serial which also
delimit the =
start of the<br>
> =A0 =A0 =A0 =A0 message - you don't<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > need this
'hack' if you foll=
ow the original spec for<br>
> =A0 =A0 =A0 =A0 non-UDP<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 transports.<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > Patrick<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > 2009/9/13 Kevin Hawkins
<<a href=
=3D"mailto:yahoogroupskh@xxxxxxx";>yahoogroupskh@xxxxxxx</a><b=
r>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@googlemail.=
com">yahoogroupskh@xxxxxxx</a>><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 <mailto:<a
href=3D"mailto:yahoogroupsk=
h@xxxxxxx">yahoogroupskh@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@googlemail.=
com">yahoogroupskh@xxxxxxx</a>>><br>
><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > <mailto:<a
href=3D"mailto:yahoogr=
oupskh@xxxxxxx">yahoogroupskh@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@googlemail.=
com">yahoogroupskh@xxxxxxx</a>><br>
><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 <mailto:<a
href=3D"mailto:yahoogroupsk=
h@xxxxxxx">yahoogroupskh@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@googlemail.=
com">yahoogroupskh@xxxxxxx</a>>>>><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 =A0 We have been
asked on se=
veral occasions how to<br>
> =A0 =A0 =A0 =A0 detect the<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 end of a<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 xAP messager as
there is no =
unique EOM character.<br>
> =A0 =A0 =A0 =A0 =A0Typically<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 in any<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 reasonable sized
packet stru=
ctured transport eg<br>
> =A0 =A0 =A0 =A0 UDP then the<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 packet<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 provides such an
indication =
but on systems with<br>
> =A0 =A0 =A0 =A0 small packets or<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 non eg<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 asynchronous
serial this is =
not useable.<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 =A0 In
discussing this with =
the specification team<br>
> =A0 =A0 =A0 =A0 we must<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 consider<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 backwards
compatability and =
so we do not wish to<br>
> =A0 =A0 =A0 =A0 alter the<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0
specification<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 to include a
unique EOM char=
acter. =A0What we do propose<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 however is that<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 xAP v1.3 will
specify that a=
ll messages should end<br>
> =A0 =A0 =A0 =A0 with two<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0
consecutive<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0
chr(10)'s immediately af=
ter the closing '}'<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 ie =A0..... 7D
0A 0A<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 =A0Some apps,
even v1.2 ones=
, =A0already do this. =A0We<br>
> =A0 =A0 =A0 =A0 don't<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 believe this<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 will cause any
backwards com=
patibility issues and<br>
> =A0 =A0 =A0 =A0 it will<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 always be<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 unique within a
xAP message.=
<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 =A0So, in
developing xAP v1.=
3 apps could you<br>
> =A0 =A0 =A0 =A0 therefore append<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 =A0two 0A's<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 at the end of
your messages =
, and of course handle<br>
> =A0 =A0 =A0 =A0 incoming<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 messages<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 containing
such.<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 =A0 K<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0
----------------------------=
--------<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0 Yahoo! Groups
Links<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 =A0mailto:<a href=3D"mailto:xAP_developer-fullfeatured=
@yahoogroups.com">xAP_developer-fullfeatured@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 <mailto:<a
href=3D"mailto:xAP_develope=
r-fullfeatured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx<=
/a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>>><b=
r>
> =A0 =A0 =A0 =A0 > =A0 =A0 > =A0 =A0
<mailto:<a href=3D"mailto=
:xAP_developer-fullfeatured@xxxxxxx">xAP_developer-fullfeatured@yah=
oogroups.com</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 <mailto:<a
href=3D"mailto:xAP_develope=
r-fullfeatured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx<=
/a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>>>&g=
t;<br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0
------------------------------------<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 Yahoo! Groups Links<br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 =A0 =A0mailto:<a
href=3D"mailto:xAP_devel=
oper-fullfeatured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx=
om</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>><br>
> =A0 =A0 =A0 =A0 > =A0 =A0 <mailto:<a
href=3D"mailto:xAP_develope=
r-fullfeatured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx<=
/a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>>><b=
r>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
> =A0 =A0 =A0 =A0 ><br>
><br>
><br>
><br>
> =A0 =A0 =A0 =A0 ------------------------------------<br>
><br>
> =A0 =A0 =A0 =A0 Yahoo! Groups Links<br>
><br>
><br>
> =A0 =A0 =A0 =A0 =A0 =A0mailto:<a href=3D"mailto:xAP_developer-fullfeat=
ured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a><br>
> =A0 =A0 =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeature=
d@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a>><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
<br>
<br>
<br>
------------------------------------<br>
<br>
xAP_Development Main Index |
xAP_Development Thread Index |
xAP_Development Home |
Archives Home
|