Re: xAP v1.3

[Kevin Hawkins <yahoogroupskh@xxxxxxxxxxxxxx>]

Hi Neil,

Glad it's all progressing. I think this sort of issue does need a
solution so glad you're tackling it. A couple of thoughts.

I would try and avoid actually altering the xAPBSC schema messages if
possible eg by adding extra parameters as that will create a non
standard implementation but you should I think be able to do what you
want without that anyway - by just creating extra BSC endpoints for the
WatchDog_Timeout_Enable and others and then setting them directly using
a second block in the same message. The xAP spec does however allow you
to include extra parameters should you wish.

In your system the constant 'tickle' by the PC does reaffirm the state
but what would happen if another controller changed the state of the
device ? Then the originating controller would change it back again. Now
you could monitor all 'event' messages from that endpoint to recognise
that another device had originated a change I suppose , or indeed all
messages targeted at that device, and then backoff as the owning
controller. It would require that any controller wanting to interact
with your device implemented the tickle process however. This I suppose
might mandate the xAPBSC.cmd actually have some identification within
the message that it can 'tickle' - which a dual block message would.
Might need some thinking with regard to multiple controllers and other
existing controllers being able to interact.

Re SNTP - my embedded gateway implements it but I'm not aware of a
standalone xAP app - seems like an obvious one for someone to write -
and there is an existing xAP date/time schema.

Being able to configure BSC devices (likely using a schema similar to
BSC) is something we really need . It's also part of a much larger xAP
config need that Patrick mentioned. I have the same need in my C-Bus
gateway in terms of matching the endpoint names to those configured in
the C-Bus Toolkit software. I know someone is working on this and so
I'll ask them to contact you (and Patrick) offlist and see if we can
come up with something useful to all.


K

Neil Wrightson wrote:
>
>
> Hi,
> I'm still in the progress of creating my first BSC device and I'm
> looking at adding a few extensions to the common BSC schema.
> From an output end point perspective -
>
>   1.
>       WatchDog_Timeout_Enable : Boolean; // True = Enable Watchdog for
>       this output, False = No watchDog
>   2.
>       Watch_Dog_Timeout_Value : Byte ; // I.e. 0 or 1 for a logic
>       output or 0..255 for an analogue output
>
> From a Device perspective
>
>   1.
>       WatchDog_TimeOut_Interval : Byte ; // 0..255 Minutes
>   2.
>       Restore_State_Value : Byte ; // 0..255 Minutes
>
> My thoughts in this area are.
> A xAP controlling PC (else where on the network) sets an output on the
> above device&endpoint.
> The PC continues to set this output via the same command at an
> interval less then the WatchDog_TimeOut_Interval.
> When the device receives an output command it records the current
> state of each watchdogged output i.e. 0,1 or 0..255 in non volatile
> memory and also a Device comms timestamp.
> If the PC stops sending the output messages, the watchdog times out.
> The device then sets the output to the Watch_Dog_Timeout_Value (This
> may actually be to an output ON i.e. a warning light).
> If there has been a power failure and the device powers up again. The
> device checks the current time against the last comms timestamp. If
> the value is less then Restore_State_Value then the outputs are set to
> the state prior to the power failure. If the time difference is great
> then set the outputs to the Watch_Dog_Timeout_Value.
> The device could either get the time from an internal RTC or an
> external SNTP server (or xAP SNTP Server? Does anybody have one of
> these?).
> I'm also looking at protocols to allow devices and endpoints to be
> dynamically assigned with names.
>
> Regards,
>
> *Neil Wrightson.*
> Skype : Neil_Wrightson
>
>
>
------------------------------------------------------------------------
> *From:* xAP_developer@xxxxxxx
> [mailto:xAP_developer@xxxxxxx] *On
Behalf Of *Edward Pearson
> *Sent:* Thursday, 17 September 2009 9:12 AM
> *To:* xAP_developer@xxxxxxx
> *Subject:* RE: [xAP_developer] xAP EOM identifier in xAP v1.3
>
> Sounds like a lot of work. I?d be interested in starting that
> discussion (as a new thread). I don?t think it?s at all a high
> priority as I?m sure IPv4 will be available for quite a few years more
> (and particularly in domestic environments). I wouldn?t want it to
> hold up v1.3 (which is ready to go aside from this EOM business). TCP
> can wait for v1.4; IPv6 probably is further out than that.
>
> My knowledge of IPv6 is superficial so I?d like to find out more. It
> has a multicast but no broadcast ? that?s about all I know (oh and
> very long addresses!).
>
> *From:* xAP_developer@xxxxxxx
> [mailto:xAP_developer@xxxxxxx] *On
Behalf Of *Bauer, Steven J.
> *Sent:* 15 September 2009 22:53
> *To:* xAP_developer@xxxxxxx
> *Subject:* RE: [xAP_developer] xAP EOM identifier in xAP v1.3
>
> If you are going work at defining the spec to be able to use tcp you
> might as well define it so that it works in the ipv6 world as well.
>
> Steve
>
> *From:* xAP_developer@xxxxxxx
> [mailto:xAP_developer@xxxxxxx] *On
Behalf Of *Edward Pearson
> *Sent:* Tuesday, September 15, 2009 3:21 PM
> *To:* xAP_developer@xxxxxxx
> *Subject:* RE: [xAP_developer] xAP EOM identifier in xAP v1.3
>
> I don?t mean that packet fragmentation or re-ordering are myths ? just
> that when dealing with xAP at the IP stack (socket) interface you are
> dealing with datagrams not data packets so you don?t need to worry
> about those packet aspects. There was a design decision to limit xAP
> datagrams to 1500 bytes to improve the likely coverage of correctly
> sent datagrams in a world of IP stacks of varying quality. It?s no
> more important than that. Readers of the spec seem to attach more
> importance to it than it needs (there?s the myth aspect). To program
> xAP you don?t need to worry about fragmentation and reordering ? just
> keep your datagrams 1500 bytes or less and let the stack do its thing.
>
> Sometimes an app goes a bit wild (xAP news has occasionally been a
> useful test source) and big xAP messages are generated ? and they get
> delivered too! It?s normally the input buffer of the receiving app (eg
> hub) that breaks first.
>
> On the tcp framing, I?d suggest that implementing the CRC part
> (irrelevant on a reliable stream) is a waste of most people?s time; by
> far more use will be made of tcp than any kind of serial/485/etc
> networks so they?d be sharing development of an implementation with
> nobody.
>
> Discovery can be done simply with the existing hub heartbeat message.
> Just need to agree on an extra block that advertises the port number
> of the tcp service ? which I assume, by default would be 3639.
>
> Having read the 802.11 spec, I now understand why broadcast udp from
> an AP to a NIC is so un-reliable. And ad-hoc mode is a real disaster!
>
> *From:* xAP_developer@xxxxxxx
> [mailto:xAP_developer@xxxxxxx] *On
Behalf Of *Patrick Lidstone
> *Sent:* 15 September 2009 14:15
> *To:* xAP_developer@xxxxxxx
> *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP v1.3
>
> Hi Edward,
> Please see in-line responses...
>
> 2009/9/15 Edward Pearson <edward.mailgroup@xxxxxxx
> <mailto:edward.mailgroup@xxxxxxx>>
>
> The double 0x0a terminator works for me, it?s simple to implement ?
> already done for my stuff.
>
> ...but it isn't robust unless the message is sent as a single
> datagram, and if it is sent as a single datagram, the os should
> deliver it as such -- and if it doesn't, adding an eom marker isn't
> going to help.
>
>     For reliable streams, such as TCP, I generally frame the messages
>     with STX and ETX.
>
>     I thought all this business about 1500 bytes was somewhat urban
>     myth (at least as it applies to xAP). The data packet size is not
>     what?s important; it?s how the particular IP stack implementation
>     deals with >datagrams< that?s the key.
>
>
> I don't understand what you are syaing here? The data packet size is
> inextricably linked to the IP stack. What aspect is it that you
> consider to be an urban myth?
>
>     I only have experience of the two most recent Windows stacks (XP
>     and Vista) which I agree are likely more capable than old embedded
>     stacks. My experiments with Wireshark show that those stacks will
>     happily deal with fragmentation and defragmentation. 64KB in a
>     single datagram? No problem if your receive buffer is long enough
>     (even if you force a small MTU).
>
> Yes, agreed, OSes perform fragmentation for you. The individual
> fragments have a maximum size as determined by the MTU. For pragmatic,
> practical reasons, xAP needs to define a maximum overall message size,
> and for convenience's sake that was set as equal to the 'standard' MTU
> size. Devices which use a smaller MTU /should /fragment and reassemble
> seamlessly provided that the correct socket options have been set to
> define the maximum UDP packet size. By electing to use a single MTU's
> worth of data, we we avoid the overhead associated with fragmentation
> and reassembly (principally memory buffering) which is a good thing.
> When reassembled, the os should deliver a datagram as a complete
> entity in one go (irrespective of the mtu size, assuming that the
> datagram falls within the maximum datagram size defined for the
> stack). If the sender doesn't send a message as a single datagram,
> then the whole thing falls apart because effectively you are then
> doing fragmentation and reassembly at the application layer, and that
> won't work because the ordering across datagrams is not guaranteed and
> individual datagrams may get lost.
>
>     If anything goes wrong (eg, missing fragment) the whole datagram
>     is dropped. I can?t see how packet re-ordering can happen on a
>     single LAN for broadcast UDP ? there are no multiple paths and no
>     retry mechanism. Certainly never observed it ? I assume the stack
>     would again just drop the entire datagram.
>
>
> Re-ordering of datagrams can occur for multiple reasons on a single
> lan. A sender may send individual UDP packets in any order it chooses.
> This commonly occurs with fragmented packets originating from a
> multi-threaded sender, which can be interleaved with smaller,
> non-fragmented datagrams as required (optimising transient memory use,
> as soon as they are sent, the buffer can be released). A switch is not
> obliged to retransmit packets in the order in which they are received.
> And most fundamentally, the specs do not require UDP packets to be
> ordered so, even if you don't observe it, it /can/ happen, and sooner
> or later interoperability issues will arise if the working assumption
> is made that they always arrive in order.
>
>     A bigger issue for me, and the reason for me experimenting a few
>     times with TCP stream serving from hubs, is datagram loss over
>     WiFi networks. This is greatly accentuated for UDP when you use
>     broadcast (as we do) from wired to wireless (fine the other way as
>     it?s not really a broadcast packet till it gets to the AP) as the
>     access point and NIC can not longer apply their ack/nak procedure
>     at the transport level. I commonly see xAP datagram loss rates
>     from wired to wireless connections of 20%. So I?d like us to agree
>     on a standard transport wrapper for TCP streams which a lot of
>     platforms would find useful.
>
> I'd suggest using the same framing as the "transport
wrapper", as this
> then allows for code to be shared across transports. If xAP was
> extended to support TCP, then that should also include a formal
> discovery mechanism by which the IP address/characteristics of the hub
> can be discovered (over UDP xAP?)
>
> Patrick
>
>     *From:* xAP_developer@xxxxxxx
>     <mailto:xAP_developer@xxxxxxx>
>     [mailto:xAP_developer@xxxxxxx
>     <mailto:xAP_developer@xxxxxxx>]
*On Behalf Of *Patrick
>     Lidstone
>     *Sent:* 14 September 2009 14:19
>     *To:* xAP_developer@xxxxxxx
>     <mailto:xAP_developer@xxxxxxx>
>     *Subject:* Re: [xAP_developer] xAP EOM identifier in xAP v1.3
>
>     So the xAP delimiters for serial are defined in the 1.2 xAP spec
here:
>     http://www.xapautomation.org/index.php?title=Protocol_definition#Transport_Wrapper
>     <http://www.xapautomation.org/index.php?title=Protocol_definition#Transport_Wrapper>
>
>     To the best of my knowledge, the MTU size for an ethernet frame is
>     1518 bytes, which leads to a UDP packet MTU of 1500 bytes and this
>     is the size that is adopted by the majority of operating systems.
>     Internet routers (i.e. ISPs) sometimes use an MTU of 576 bytes,
>     but this wouldn't be relevant to xAP since the traffic doesn't
>     pass over the wider net, or if it does, it's generally gatewayed
>     via a TCP/IP connection.
>
>     If a device is receiving fragmented UDP packets, I think the same
>     issues arise as those related to extending the xAP message size
>     beyond 1500 bytes - what happens if a fragment gets discarded.
>
>     If you take the scenario:
>
>     Part 1(a), Part 1(b), Part 2(a), Part 2(b), Part 3(a), Part 3(b)
>
>     --- first there is no guarantee that the parts will be delivered
>     in order, and second, if part 1(b) was dropped, and you were
>     blindly assembling messages based on the proposed double-0a
>     terminator, you'd end up with a message comprising part 1(a), 2(a)
>     and 2(b) which is not only obviously corrupt, but also possibly
>     larger than the maximum xAP message size, blowing away your
buffers.
>
>     I think the solution is to probably parse incoming messages on the
>     fly, byte-by-byte. You can then at least reset your state when you
>     encounter the xap-header, and if you count open and close curly
>     braces, you can tell when you have an apparently complete message.
>     This won't solve the issue of UDP fragments being potentially
>     received out of order, but so long as we are dealing with a single
>     LAN, and fragmentation occurs at the receiver, we will be ok I
think.
>
>     It is absolutely possible for UDP packets to be discarded, and the
>     way we deal with this in xAP is to accept that this can happen to
>     xAP messages, and layer application level acknowledgements where
>     knowing that a message has been received is critically important -
>     whether explicitly or implicitly through status messages. There
>     are various schemes that could be adopted to allow a receiver to
>     detect lost messages (e.g. sequence numbering), but they quickly
>     become quite onerous, and assume that the originator keeps a copy
>     of the original message or is able to reconstruct it - which is
>     non-trivial for the many xAP nodes.
>
>     Perhaps you can share more of the specific details of the
>     device(s) in question (manufacturer, docs, o/s etc), and their
>     specific behaviour, which seems a bit anomalous?
>
>     Patrick
>
>     2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>>
>
>     One of the issues seems to be that there is conflicting views as
>     to the
>     length of a a UDP data packet payload. Some people cite 500 or 512
>     characters and some 1500. Regardless in some low
memory/performance
>     devices it is being reported, that even with UDP, packets are
being
>     received from the buffer either truncated or appended back to
back.
>     The latter I assume is due to the speed of the device in servicing
the
>     buffer.
>
>     We have an opportunity to protect against this with v1.3 and the
>     double
>     '0A' seems the most compatible. I would be loathe to support
anything
>     that wasn't backward compatible.
>
>     K
>
>
>     Patrick Lidstone wrote:
>     >
>     >
>     > I will dig it out - it included an optional checksum I think,
>     and IIRC
>     > was framed by stx and etx (a kind of pseudo industry
standard). I
>     > certainly used it with the PIC serial stuff and the
xAP-serial
>     bridge.
>     > Re.: long message truncation and concatenation: If we need to
>     support
>     > messages that are larger than one UDP packet, then there are
>     > additional complexities and the proposed scheme won't work as
>     intended
>     > as the ordering of UDP messages is not guaranteed. I'm happy
to help
>     > refine a spec to support these capabilities, but it is moving
away
>     > from the basic ethos of xAP somewhat, as devices would have
to
>     be able
>     > to buffer received messages, and that raises the bar
considerably.
>     > Perhaps there is scope for co-existence of a long and
standard
>     message
>     > protocol though?
>     >
>     > Patrick
>     >
>     > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>
>
>     > <mailto:yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>>>
>
>     >
>     > ... Oh ... where is that in the spec ? it might be all we
need.
>     >
>     > This is also tied in with some aspects of long message
truncation
>     > and concatenation of messages received in UDP receive buffers
>     > though...
>     >
>     > K
>     >
>     > Patrick Lidstone wrote:
>     > >
>     > >
>     > > The original xap spec provides extensions for framing a
>     message over
>     > > async serial which also delimit the start of the message
- you
>     don't
>     > > need this 'hack' if you follow the original spec for
non-UDP
>     > transports.
>     > >
>     > > Patrick
>     > >
>     > > 2009/9/13 Kevin Hawkins <yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>
>     > <mailto:yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>>
>
>     > > <mailto:yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>
>
>     > <mailto:yahoogroupskh@xxxxxxx
>     <mailto:yahoogroupskh@xxxxxxx>>>>
>     > >
>     > > We have been asked on several occasions how to detect
the
>     > end of a
>     > > xAP messager as there is no unique EOM character.
Typically
>     > in any
>     > > reasonable sized packet structured transport eg UDP then
the
>     > packet
>     > > provides such an indication but on systems with small
packets or
>     > > non eg
>     > > asynchronous serial this is not useable.
>     > >
>     > > In discussing this with the specification team we must
>     > consider
>     > > backwards compatability and so we do not wish to alter
the
>     > > specification
>     > > to include a unique EOM character. What we do propose
>     > however is that
>     > > xAP v1.3 will specify that all messages should end with
two
>     > > consecutive
>     > > chr(10)'s immediately after the closing '}'
>     > >
>     > > ie ..... 7D 0A 0A
>     > >
>     > > Some apps, even v1.2 ones, already do this. We don't
>     > believe this
>     > > will cause any backwards compatibility issues and it
will
>     > always be
>     > > unique within a xAP message.
>     > >
>     > > So, in developing xAP v1.3 apps could you therefore
append
>     > two 0A's
>     > > at the end of your messages , and of course handle
incoming
>     > messages
>     > > containing such.
>     > >
>     > > K
>     > >
>     > >
>     > > ------------------------------------
>     > >
>     > > Yahoo! Groups Links
>     > >
>     > >
>     > > mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>
>     > <mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>>
>     > > <mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>
>     > <mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>>>
>     > >
>     > >
>     > >
>     > >
>     > >
>     > >
>     >
>     >
>     >
>     > ------------------------------------
>     >
>     > Yahoo! Groups Links
>     >
>     >
>     > mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>
>     > <mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>>
>     >
>     >
>     >
>     >
>     >
>     >
>
>
>
>     ------------------------------------
>
>     Yahoo! Groups Links
>
>
>     (Yahoo! ID required)
>
>     mailto:xAP_developer-fullfeatured@xxxxxxx
>     <mailto:xAP_developer-fullfeatured@xxxxxxx>
>
>
>
>
>



------------------------------------