[Message Prev][Message
Next][Thread Prev][Thread Next][Message
Index][Thread Index]
Re: xAP EOM identifier in xAP v1.3
--001485eb04fd7ce850047389822d
Content-Type: text/plain; charset=ISO-8859-1
So the xAP delimiters for serial are defined in the 1.2 xAP spec here:
http://www.xapautomation.org/index.php?title=Protocol_definition#Transport_Wrapper
To the best of my knowledge, the MTU size for an ethernet frame is 1518
bytes, which leads to a UDP packet MTU of 1500 bytes and this is the size
that is adopted by the majority of operating systems. Internet routers
(i.e.
ISPs) sometimes use an MTU of 576 bytes, but this wouldn't be relevant to
xAP since the traffic doesn't pass over the wider net, or if it does, it's
generally gatewayed via a TCP/IP connection.
If a device is receiving fragmented UDP packets, I think the same issues
arise as those related to extending the xAP message size beyond 1500 bytes
-
what happens if a fragment gets discarded.
If you take the scenario:
Part 1(a), Part 1(b), Part 2(a), Part 2(b), Part 3(a), Part 3(b)
--- first there is no guarantee that the parts will be delivered in order,
and second, if part 1(b) was dropped, and you were blindly assembling
messages based on the proposed double-0a terminator, you'd end up with a
message comprising part 1(a), 2(a) and 2(b) which is not only obviously
corrupt, but also possibly larger than the maximum xAP message size,
blowing
away your buffers.
I think the solution is to probably parse incoming messages on the fly,
byte-by-byte. You can then at least reset your state when you encounter the
xap-header, and if you count open and close curly braces, you can tell when
you have an apparently complete message. This won't solve the issue of UDP
fragments being potentially received out of order, but so long as we are
dealing with a single LAN, and fragmentation occurs at the receiver, we
will
be ok I think.
It is absolutely possible for UDP packets to be discarded, and the way we
deal with this in xAP is to accept that this can happen to xAP messages,
and
layer application level acknowledgements where knowing that a message has
been received is critically important - whether explicitly or implicitly
through status messages. There are various schemes that could be adopted to
allow a receiver to detect lost messages (e.g. sequence numbering), but
they
quickly become quite onerous, and assume that the originator keeps a copy
of
the original message or is able to reconstruct it - which is non-trivial
for
the many xAP nodes.
Perhaps you can share more of the specific details of the device(s) in
question (manufacturer, docs, o/s etc), and their specific behaviour, which
seems a bit anomalous?
Patrick
2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx>
> One of the issues seems to be that there is conflicting views as to
the
> length of a a UDP data packet payload. Some people cite 500 or 512
> characters and some 1500. Regardless in some low memory/performance
> devices it is being reported, that even with UDP, packets are being
> received from the buffer either truncated or appended back to back.
> The latter I assume is due to the speed of the device in servicing the
> buffer.
>
> We have an opportunity to protect against this with v1.3 and the
double
> '0A' seems the most compatible. I would be loathe to support
anything
> that wasn't backward compatible.
>
> K
>
> Patrick Lidstone wrote:
> >
> >
> > I will dig it out - it included an optional checksum I think, and
IIRC
> > was framed by stx and etx (a kind of pseudo industry standard). I
> > certainly used it with the PIC serial stuff and the xAP-serial
bridge.
> > Re.: long message truncation and concatenation: If we need to
support
> > messages that are larger than one UDP packet, then there are
> > additional complexities and the proposed scheme won't work as
intended
> > as the ordering of UDP messages is not guaranteed. I'm happy to
help
> > refine a spec to support these capabilities, but it is moving
away
> > from the basic ethos of xAP somewhat, as devices would have to be
able
> > to buffer received messages, and that raises the bar
considerably.
> > Perhaps there is scope for co-existence of a long and standard
message
> > protocol though?
> >
> > Patrick
> >
> > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>>
> >
> > ... Oh ... where is that in the spec ? it might be all we
need.
> >
> > This is also tied in with some aspects of long message
truncation
> > and concatenation of messages received in UDP receive buffers
> > though...
> >
> > K
> >
> > Patrick Lidstone wrote:
> > >
> > >
> > > The original xap spec provides extensions for framing a
message
> over
> > > async serial which also delimit the start of the message
- you
> don't
> > > need this 'hack' if you follow the original spec for
non-UDP
> > transports.
> > >
> > > Patrick
> > >
> > > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>
> > > <mailto:yahoogroupskh@xxxxxxx
> > <mailto:yahoogroupskh@xxxxxxx>>>
> > >
> > > We have been asked on several occasions how to
detect the
> > end of a
> > > xAP messager as there is no unique EOM character.
Typically
> > in any
> > > reasonable sized packet structured transport eg UDP
then the
> > packet
> > > provides such an indication but on systems with
small packets
> or
> > > non eg
> > > asynchronous serial this is not useable.
> > >
> > > In discussing this with the specification team we
must
> > consider
> > > backwards compatability and so we do not wish to
alter the
> > > specification
> > > to include a unique EOM character. What we do
propose
> > however is that
> > > xAP v1.3 will specify that all messages should end
with two
> > > consecutive
> > > chr(10)'s immediately after the closing '}'
> > >
> > > ie ..... 7D 0A 0A
> > >
> > > Some apps, even v1.2 ones, already do this. We
don't
> > believe this
> > > will cause any backwards compatibility issues and it
will
> > always be
> > > unique within a xAP message.
> > >
> > > So, in developing xAP v1.3 apps could you therefore
append
> > two 0A's
> > > at the end of your messages , and of course handle
incoming
> > messages
> > > containing such.
> > >
> > > K
> > >
> > >
> > > ------------------------------------
> > >
> > > Yahoo! Groups Links
> > >
> > >
> > > mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>
> > > <mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>>
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> > mailto:xAP_developer-fullfeatured@xxxxxxx
> > <mailto:xAP_developer-fullfeatured@xxxxxxx>
> >
> >
> >
> >
> >
> >
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>
>
--001485eb04fd7ce850047389822d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
<head>
<style type=3D"text/css">
<!--
/* start of attachment style */
.ygrp-photo-title{
clear: both;
font-size: smaller;
height: 15px;
overflow: hidden;
text-align: center;
width: 75px;
}
div.ygrp-photo{
background-position: center;
background-repeat: no-repeat;
background-color: white;
border: 1px solid black;
height: 62px;
width: 62px;
}
div.photo-title=20
a,
div.photo-title a:active,
div.photo-title a:hover,
div.photo-title a:visited {
text-decoration: none;=20
}
div.attach-table div.attach-row {
clear: both;
}
div.attach-table div.attach-row div {
float: left;
/* margin: 2px;*/
}
p {
clear: both;
padding: 15px 0 3px 0;
overflow: hidden;
}
div.ygrp-file {
width: 30px;
valign: middle;
}
div.attach-table div.attach-row div div a {
text-decoration: none;
}
div.attach-table div.attach-row div div span {
font-weight: normal;
}
div.ygrp-file-title {
font-weight: bold;
}
/* end of attachment style */
-->
</style>
</head>
<html><head>
<style type=3D"text/css">
<!--
#ygrp-mkp{
border: 1px solid #d8d8d8;
font-family: Arial;
margin: 14px 0px;
padding: 0px 14px;
}
#ygrp-mkp hr{
border: 1px solid #d8d8d8;
}
#ygrp-mkp #hd{
color: #628c2a;
font-size: 85%;
font-weight: bold;
line-height: 122%;
margin: 10px 0px;
}
#ygrp-mkp #ads{
margin-bottom: 10px;
}
#ygrp-mkp .ad{
padding: 0 0;
}
#ygrp-mkp .ad a{
color: #0000ff;
text-decoration: none;
}
-->
</style>
</head>
<body>
<!-- **begin egp html banner** -->
<br><br>
<!-- **end egp html banner** -->
So the xAP delimiters for serial are defined in the 1.2 xAP spec
here:<br><=
a href=3D"http://www.xapautomation.org/index.php?title=3DProtocol_definitio=
n#Transport_Wrapper">http://www.xapautomation.org/index.php?title=3DProtoco=
l_definition#Transport_Wrapper</a><br>
<br>To the best of my knowledge, the MTU size for an ethernet frame
is 1518=
bytes, which leads to a UDP packet MTU of 1500 bytes and this is the size =
that is adopted by the majority of operating systems. Internet routers
(i.e=
. ISPs) sometimes use an MTU of 576 bytes, but this wouldn't be
relevan=
t to xAP since the traffic doesn't pass over the wider net, or if
it do=
es, it's generally gatewayed via a TCP/IP connection.<br>
<br>If a device is receiving fragmented UDP packets, I think the same
issue=
s arise as those related to extending the xAP message size beyond 1500
byte=
s - what happens if a fragment gets discarded.<br><br>If you
take the scena=
rio:<br>
<br>Part 1(a), Part 1(b), Part 2(a), Part 2(b), Part 3(a), Part
3(b)<br><br=
>--- first there is no guarantee that the parts will be delivered in
order,=
and second, if part 1(b) was dropped, and you were blindly assembling mess=
ages based on the proposed double-0a terminator, you'd end up with
a me=
ssage comprising part 1(a), 2(a) and 2(b) which is not only obviously
corru=
pt, but also possibly larger than the maximum xAP message size, blowing
awa=
y your buffers.<br>
<br>I think the solution is to probably parse incoming messages on
the fly,=
byte-by-byte. You can then at least reset your state when you encounter th=
e xap-header, and if you count open and close curly braces, you can tell
wh=
en you have an apparently complete message. This won't solve the
issue =
of UDP fragments being potentially received out of order, but so long as
we=
are dealing with a single LAN, and fragmentation occurs at the receiver, w=
e will be ok I think.<br>
<br>It is absolutely possible for UDP packets to be discarded, and
the way =
we deal with this in xAP is to accept that this can happen to xAP
messages,=
and layer application level acknowledgements where knowing that a message =
has been received is critically important - whether explicitly or
implicitl=
y through status messages. There are various schemes that could be adopted
=
to allow a receiver to detect lost messages (e.g. sequence numbering), but
=
they quickly become quite onerous, and assume that the originator keeps a
c=
opy of the original message or is able to reconstruct it - which is
non-tri=
vial for the many xAP nodes.<br>
<br>Perhaps you can share more of the specific details of the
device(s) in =
question (manufacturer, docs, o/s etc), and their specific behaviour,
which=
seems a bit
anomalous?<br><br>Patrick<br><br><div
class=3D"gmail_quote">
2009/9/13 Kevin Hawkins <span dir=3D"ltr"><<a
href=3D"mailto:yahoogroups=
kh@xxxxxxx">yahoogroupskh@xxxxxxx</a>></span><br><blockquo=
te class=3D"gmail_quote" style=3D"border-left: 1px solid
rgb(204, 204, 204)=
; margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
One of the issues seems to be that there is conflicting views as to
the<br>
length of a a UDP data packet payload. =A0Some people cite 500 or
512<br>
characters and some 1500. =A0Regardless in some low
memory/performance<br>
devices it is being reported, that even with UDP, =A0packets are
being<br>
received from the buffer either truncated or appended back to
back.<br>
The latter I assume is due to the speed of the device in servicing
the<br>
buffer.<br>
<br>
We have an opportunity to protect against this with v1.3 and the
double<br>
'0A' seems the most compatible. =A0 I would be loathe to
support an=
ything<br>
that wasn't backward compatible.<br>
<br>
=A0 K<br>
<div class=3D"im"><br>
Patrick Lidstone wrote:<br>
><br>
><br>
> I will dig it out - it included an optional checksum I think, and
IIRC=
<br>
> was framed by stx and etx (a kind of pseudo industry standard).
I<br>
> certainly used it with the PIC serial stuff and the xAP-serial
bridge.=
<br>
> Re.: long message truncation and concatenation: If we need to
support<=
br>
> messages that are larger than one UDP packet, then there
are<br>
> additional complexities and the proposed scheme won't work
as inte=
nded<br>
> as the ordering of UDP messages is not guaranteed. I'm
happy to he=
lp<br>
> refine a spec to support these capabilities, but it is moving
away<br>
> from the basic ethos of xAP somewhat, as devices would have to be
able=
<br>
> to buffer received messages, and that raises the bar
considerably.<br>
> Perhaps there is scope for co-existence of a long and standard
message=
<br>
> protocol though?<br>
><br>
> Patrick<br>
><br>
> 2009/9/13 Kevin Hawkins <<a href=3D"mailto:yahoogroupskh@googlemail=
.com">yahoogroupskh@xxxxxxx</a><br>
</div>> <mailto:<a href=3D"mailto:yahoogroupskh@xxxxxxx";>yahoo=
groupskh@xxxxxxx</a>>><br>
<div class=3D"im">><br>
> =A0 =A0 ... Oh ... where is that in the spec ? =A0it might be all
we n=
eed.<br>
><br>
> =A0 =A0 =A0 =A0This is also tied in with some aspects of long
message =
truncation<br>
> =A0 =A0 and concatenation of messages received in UDP receive
buffers<=
br>
> =A0 =A0 though...<br>
><br>
> =A0 =A0 =A0K<br>
><br>
> =A0 =A0 Patrick Lidstone wrote:<br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 > The original xap spec provides extensions for
framing a m=
essage over<br>
> =A0 =A0 > async serial which also delimit the start of the
message =
- you don't<br>
> =A0 =A0 > need this 'hack' if you follow
the original spec =
for non-UDP<br>
> =A0 =A0 transports.<br>
> =A0 =A0 ><br>
> =A0 =A0 > Patrick<br>
> =A0 =A0 ><br>
> =A0 =A0 > 2009/9/13 Kevin Hawkins <<a
href=3D"mailto:yahoogroups=
kh@xxxxxxx">yahoogroupskh@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:yahoogroupskh@xxxxxxx";>yah=
oogroupskh@xxxxxxx</a>><br>
</div>> =A0 =A0 > <mailto:<a href=3D"mailto:yahoogroupskh@googlema=
il.com">yahoogroupskh@xxxxxxx</a><br>
<div><div></div><div
class=3D"h5">> =A0 =A0 <mailto:<a
href=3D"mailto=
:yahoogroupskh@xxxxxxx">yahoogroupskh@xxxxxxx</a>>>>=
<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 =A0 We have been asked on several
occasions how t=
o detect the<br>
> =A0 =A0 end of a<br>
> =A0 =A0 > =A0 =A0 xAP messager as there is no unique EOM
character.=
=A0Typically<br>
> =A0 =A0 in any<br>
> =A0 =A0 > =A0 =A0 reasonable sized packet structured
transport eg U=
DP then the<br>
> =A0 =A0 packet<br>
> =A0 =A0 > =A0 =A0 provides such an indication but on
systems with s=
mall packets or<br>
> =A0 =A0 > =A0 =A0 non eg<br>
> =A0 =A0 > =A0 =A0 asynchronous serial this is not
useable.<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 =A0 In discussing this with the
specification tea=
m we must<br>
> =A0 =A0 consider<br>
> =A0 =A0 > =A0 =A0 backwards compatability and so we do not
wish to =
alter the<br>
> =A0 =A0 > =A0 =A0 specification<br>
> =A0 =A0 > =A0 =A0 to include a unique EOM character.
=A0What we do =
propose<br>
> =A0 =A0 however is that<br>
> =A0 =A0 > =A0 =A0 xAP v1.3 will specify that all messages
should en=
d with two<br>
> =A0 =A0 > =A0 =A0 consecutive<br>
> =A0 =A0 > =A0 =A0 chr(10)'s immediately after the
closing '=
}'<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 ie =A0..... 7D 0A 0A<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 =A0Some apps, even v1.2 ones, =A0already
do this.=
=A0We don't<br>
> =A0 =A0 believe this<br>
> =A0 =A0 > =A0 =A0 will cause any backwards compatibility
issues and=
it will<br>
> =A0 =A0 always be<br>
> =A0 =A0 > =A0 =A0 unique within a xAP message.<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 =A0So, in developing xAP v1.3 apps could
you ther=
efore append<br>
> =A0 =A0 =A0two 0A's<br>
> =A0 =A0 > =A0 =A0 at the end of your messages , and of
course handl=
e incoming<br>
> =A0 =A0 messages<br>
> =A0 =A0 > =A0 =A0 containing such.<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 =A0 K<br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0
------------------------------------<br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 Yahoo! Groups Links<br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 > =A0 =A0 =A0 =A0mailto:<a href=3D"mailto:xAP_developer-ful=
lfeatured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a><b=
r>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeatured@yahoog=
roups.com">xAP_developer-fullfeatured@xxxxxxx</a>><br>
> =A0 =A0 > =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfe=
atured@xxxxxxx">xAP_developer-fullfeatured@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeatured@yahoog=
roups.com">xAP_developer-fullfeatured@xxxxxxx</a>>><br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
> =A0 =A0 ><br>
><br>
><br>
><br>
> =A0 =A0 ------------------------------------<br>
><br>
> =A0 =A0 Yahoo! Groups Links<br>
><br>
><br>
> =A0 =A0 =A0 =A0mailto:<a href=3D"mailto:xAP_developer-fullfeatured@yah=
oogroups.com">xAP_developer-fullfeatured@xxxxxxx</a><br>
> =A0 =A0 <mailto:<a href=3D"mailto:xAP_developer-fullfeatured@yahoog=
roups.com">xAP_developer-fullfeatured@xxxxxxx</a>><br>
><br>
><br>
><br>
><br>
><br>
><br>
<br>
<br>
<br>
------------------------------------<br>
<br>
xAP_Development Main Index |
xAP_Development Thread Index |
xAP_Development Home |
Archives Home
|