next topic »

[on:]

In Arduino 1.0, the documentation for the SoftwareSerial library < http://arduino.cc/en/Reference/SoftwareSerial > needs corrections:

1)  Documentation for read() says "Reads a character from the receive pin of the software serial port. This function waits for a character to arrive, reads it, and returns the character read. Data that arrives at other times is lost."

In fact, the software serial port is "reading" all of the time (after begin(...) or listen()) and putting the data into a buffer.  read() returns the next character in the buffer, or -1 if the buffer is empty.  There is no waiting and no lost data unless the buffer is full.

2)  Documentation for overflow() neglects to mention that overflow() has the side effect of clearing the overflow flag.  If overflow() returns true, the next call to overflow() will return false unless another character arrives in the meantime.

3)  Documentation for SoftwareSerial should mention that it inherits from Serial and Print, and thus Serial and Print public methods may be used as well.

[Reply #1 on:]

Thanks for this 
I found point 1) a bit confusing so nice to have it clarified.
point 2) thats nice to know thanks.
point 3) does this mean that flush and peek can also be used (coming from Serial) not sure that Print offers

[Reply #2 on:]

flush() and peek() are already offered by SoftwareSerial, they are just not documented.  That is part of the problem.
I am new to C++ and have not tried it, but I believe that, via SoftwareSerial, Stream will provide setTimeout, two versions of find, two version of findUntil, parseInt, parseFloat, readBytes, and readBytesUntil.  Print should provide 11 versions of print and 12 versions of println.  See Stream.h and Print.h for details.

[Reply #3 on:]

Be aware that in all the last minute changes they did for 1.0 that they broke compatibility between HardwareSerial
and SoftwareSerial for flush().

So "as is" in 1.0, flush() on SoftwareSerial and HardwareSerial  do not do the same thing.
(They changed the flush() function in HardwareSerial, after they added it to SoftwareSerial)

flush() on SoftwareSerial purges all the received characters in the rx buffer.
flush() on HardwareSerial waits for all the characters in the tx buffer to be transmitted.

Pre 1.0 SoftwareSerial didn't have a flush()
Pre 1.0 HardwareSerial flush() purged the received characters.

On 1.0 there is no API call to purge the rx buffer on HardwareSerial.

IMO, a new API call should have been created to drain the tx buffer rather
than change the behavior of the exiting flush() api function.

-- bill

[Reply #4 on:]

IMO, a new API call should have been created to drain the tx buffer rather
than change the behavior of the exiting flush() api function.

I would (almost) agree. The basing on the Stream class, though, dictates that the flush() function operate in the same fashion as flush() on any stream. Typically, flushing a stream means that a send of the buffered data is forced, rather than waiting for the buffer to fill. This is now the behavior of flush() in the HardwareSerial class. The unfortunate thing is that there was already a function of that name in the HardwareSerial class, even though what it did was not what one typically expects flush to do.

If one looks at a lot of code posted on the forum, the current functionality of the flush() function is what most people expected the old functionality to do.

[Reply #5 on:]

Changing an existing API always creates pain.
But to create the same API function in two different modules that does 2 different things
in modules that are logically supposed to have the same API is, IMO, inexcusable.

So I'm not that picky on the names of functions, I am very picky that I believe that flush() should
do the same thing in HardwareSerial and SoftwareSerial.
(Maybe it's already been bugged)

The issue with this particular change/difference is that it silently breaks code with no warning.
Arduino 1.0 users can get burned when moving code between HardwareSerial and SoftwareSerial
because while the SoftwareSerial does not buffer TX characters and so a flush() call
will not affect TX output,
Code that uses flush() in an attempt to push out any buffered TX characters will break when used on SofwareSerial
as it will potentially throw away all the RX data rather than the intended function of ensuring that the TX buffer is pushed
out the port.

This is a SERIOUS problem.

On a side note:
I believe that what can add to the "flush" confusion is that different APIs
us the term "flush" differently.

termio which is now part of POSIX was used on tty serial ports even before C++ existed
"flush" means purge the data in the buffer (buffer can be rx or tx),
"drain" means push the data in the buffer out.

And it is similar in streamio which is used for sockets.

Win32 used similar meanings/definitions for "flush", and "drain" but also has a "purge" to purge buffers.

So from that "C" perspective, the old flush() functionality seemed more appropriate.
But it's only a name.

Since I'm an old C guy, from my vantage point it looks like the
the confusion comes from C++ stream/iostream picking "flush()" as the name of the function
to push out unbuffered data.
I wish that they had picked "drain()" instead as it would have been more consistent
with the existing C API terminology but it is way too late to ever change it.

So depending if you are a C person or a C++ person and which APIs you are more
accustomed to, "flush" has different meanings.

--- bill

[Reply #6 on:]

I wish that they had picked "drain()" instead as it would have been more consistent
with the existing C API terminology but it is way too late to ever change it.

None of the names (flush, drain, purge) is particularly appropriate, in my opinion.

A function that ignores the contents of a buffer ought to be called just that.

A function that commits what is in the buffer to a device should be called just that.

And, yes, SoftwareSerial and HardwareSerial should each do exactly the same thing when flush() is called.

[Reply #7 on:]

Thanks for the report.  I updated the SoftwareSerial documentation accordingly.  I added documentation for peek(), but not for flush() because of the confusion that it behaves differently than for the HardwareSerial port.

[Reply #8 on:]

Thanks for the report.  I updated the SoftwareSerial documentation accordingly.  I added documentation for peek(), but not for flush() because of the confusion that it behaves differently than for the HardwareSerial port.

My opinion is that the best way to eliminate confusion is to actually document the way things really work
rather than leave them undocumented.
Just because it is inconsistent between the two libraries should not be a reason not to document
the way things actually work for each of them.

Perhaps even document how they were intended to work, and how they may work in the future, and
any potential work arounds to deal with any inconsistencies in the interim.

--- bill

[Reply #9 on:]

My main issue with the flush command is why it's even provided in the first place? I see it used in most cases by beginners thinking its somehow required or useful for normal comm transactions, and almost always is the wrong thing to do for what they are trying to accomplish.

 I've asked out of curiosity on this forum before if anyone could state a useful reason to issue a flush command for any 'normal' type of transaction but don't recall ever getting any responses. I'm not saying it's impossible to come up with a usage example but I would like to see some and have explained why it is used and why it is required.

Not really a rant, just very curious. I worked on many minicomputer systems in the 70s, some that were handling lots of serial comm lines and never recall them having a user API to flush buffers.

Lefty

[Reply #10 on:]

My main issue with the flush command is why it's even provided in the first place?

Lefty

Unix has had this capability in the serial port drivers long before streams and long
before C++ ever existed. They are there because, both can be very useful and are actually necessary
for certain situations.
(Even Windows implemented both in its serial drivers)

The answer as to how it can be used depends on if you are talking about the pre 1.0 flush() (which flushed
the RX buffer) vs the 1.0 flush() (which drains the TX output buffer)
and I'll give answers to both because it can be useful to flush() input queues as well
as drain output queues.

Often it comes down to synchronizing between the two ends during some type of protocol.
Depending on what is going on and the protocol, it can require draining the TX buffer,
purging the RX buffer or both.
Sometimes it can affect overall throughput and latency.

First the need to flush an input queue.(purge/throw away any RX buffered bytes)

There are times when you want to start synchronization over and it is helpful
to toss everything in one clean flush.
Lets say you are debugging and want to have a diagnostic message that pushes out a message
to the user and then waits for a key to be pressed.
But you want to always ensure that the user sees the message and must press a key to
continue.
The safest way to do this is to push out your message and then flush the input Q
before you hang your read for the character to continue on.
That way you never allow any buffered RX characters to continue on.
It ensures that there will be a pause before continuing on.
In this case it is often useful to also drain out the TX side so you know when to flush the RX side.
So the normal sequence for something like this would be
- send message
- drain output buffer to ensure user can see it
- flush RX buffers to remove any keys pressed in advance or left over from last time
- read RX for "continue" key press.

This same thing happens in more complex machine to machine data protocols.
There are times when you need to ensure the state of the TX and RX buffers.

Now lets look at the output drain.

Consider a case of using something like a serial device that takes serial commands but requires
throttling (pauses) between commands.
The normal sequence would be:
- send message
- pause before next message
(repeat)

[ I have seen Arduino serial LCD backpack libraries that do this ]

Now if you don't have any TX buffering that works.
As soon as you have TX buffering, you can't do this because your pause is happening
while characters are still going out and your throttling delay will not be as long as you want/need.
While you could account for this by taking into consideration the length of the message and
the data rate of the message transfer, it easier to simply ask for the characters to be drained out
so that you know when to start your throttling delay.

These are just 2 very simple examples of why you need both RX flush and TX drain.
You also have to keep in mind that a serial read/write API can be used on top of many types
of different hardware.
Suppose it is very expensive to transmit a message and the cost is more based sending a message
vs the size of the message.
So in that case the character/message handler might queue up the characters and wait for an end of message terminator
or a certain amount of characters in the buffer before sending any of the message.
Now in the case of short message, there might be a timeout period that will push out short messages.
But if the application wants or needs to push out all the bytes sent *NOW* it could call a drain routine to force
out all the characters buffered up.
And maybe the hardware does not provide a timeout for short messages.
In that case, none of the characters in the buffer would be sent until a drain() was done.

These are the kinds of things that happen in real OS environments with higher level data protocols
and believe it or not this same kind of thing happens inside the FTDI chip.
The FTDI chip has all kinds of buffer handing parameters to optimize the data transfers across the USB.
And how you deal with this kind of stuff affects transfer rates and latencies.
It is not always optimum to push out characters as soon as they arrive.

When designing APIs it always useful to take a broader view about services needed.
And that is why it is useful to support both RX flush and TX drain as the same API
is often used on multiple different hardware implementations and you never know what
type of protocols will be built on top of the API.

It is unfortunate that Arduino does not support being able to purge out the RX Q and
drain out the TX Q and that the flush() API call works differently on 1.0 and pre 1.0
and ti works differently on 1.0 between SoftSerial and HardwareSerial.

--- bill

[Reply #11 on:]

When designing APIs it always useful to take a broader view about services needed.
And that is why it is useful to support both RX flush and TX drain as the same API
is often used on multiple different hardware implementations and you never know what
type of protocols will be built on top of the API.

It is unfortunate that Arduino does not support being able to purge out the RX Q and
drain out the TX Q and that the flush() API call works differently on 1.0 and pre 1.0
and ti works differently on 1.0 between SoftSerial and HardwareSerial.

Well I'm still not totally sold on the need for flush. As I said I worked on minicomputer systems that used 128 channel hardware based serial multiplexers the did indeed use buffers, but all hardware based with no software control on buffer purging. And those systems with proper software supported many different protocols, including Modbus, and other node based networks.

As far as taking a broad view on API services why doesn't the serial API offer even the barest of receive error detection, like parity errors, or framing errors, etc? The hardware supports it. I would think that would have a higher priority for a user API addition over silly lets let the users play with buffers.  

[Reply #12 on:]

When designing APIs it always useful to take a broader view about services needed.
And that is why it is useful to support both RX flush and TX drain as the same API
is often used on multiple different hardware implementations and you never know what
type of protocols will be built on top of the API.

It is unfortunate that Arduino does not support being able to purge out the RX Q and
drain out the TX Q and that the flush() API call works differently on 1.0 and pre 1.0
and ti works differently on 1.0 between SoftSerial and HardwareSerial.

Well I'm still not totally sold on the need for flush. As I said I worked on minicomputer systems that used 128 channel hardware based serial multiplexers the did indeed use buffers, but all hardware based with no software control on buffer purging. And those systems with proper software supported many different protocols, including Modbus, and other node based networks.

Which flush?

Yep. I also worked on a 128 port serial crossbar switch in the early 80's.
Totally different animal. In that situation it is all about moving the bytes from port to
port and not dropping anything, while still processing and handling hardware and/or software
flow control.
This is completely different from pushing data over protocols that must be able to
recover from lost bytes or do throttling on messages.
When trying to recover you sometimes have to resynchronize the two ends
or deal with buffering and end to end latency issues and that is where having the ability
to drain and flush buffers can become important.

When throttling messages, you have to know when the last character of
a message is actually sent (not buffered) in order to hold off the first character of the next message
the proper amount of time.

As far as taking a broad view on API services why doesn't the serial API offer even the barest of receive error detection, like parity errors, or framing errors, etc? The hardware supports it. I would think that would have a higher priority for a user API addition over silly lets let the users play with buffers.  

While it might be useful to have more information like that, from a protocol perspective,
dealing output draining is sometimes more important than knowing about
RX type errors.

Consider the real world case I outlined above of a serial LCD backpack on arduino
that needs delays between serial messages.
It can become messy and complicated if the library/driver interface can not tell you when the
last character was actually transmitted.
The flush() on Arduino 1.0 solves that problem.
When it returns, you know that all the characters written to the serial device
have been transmitted
and you can start your throttling delay to determine when it is safe to
send the next message.

Think broader. The same read()/write() interface should work across all kinds of devices.
What happens when data gets buffered up to a message size but doesn't timeout out or
has a timeout of several seconds if the message is shorter than the message size?
Suppose you don't want to wait seconds for that message to be transmitted.
That is the case when you would call a "drain" routine to kick/force the characters out the device.

I've been doing serial stuff off and on for 30+ years and I can say that I have used
flush and drain off and on through the years as well.
From the early Apple ][ days, implementing bell 202 half duplex modem file transfer support,
file transfer and terminal emulation support on the early IBM PC,
GDBmon serial support on 68k processors in the late 80s,
Protocol processing over hundreds of modems for the 711 Movie Quick system in the late 80s,
to recent Scuba Dive Computer serial data transfers on Windows and Linux,
to debugging on Arduino.

Consider this oddball case that I have used in the past on some systems when you had
no access to short timers (this was 25+ years ago)
You can use the UART to time a delay. If you set the baud rate appropriately, you send
characters out the serial port to time a delay. When the transmission is done, you know how much time has elapsed.
If the characters are buffered, you need a way to drain the buffer and know when the all the
data has been transmitted. Otherwise your timing will be way too short.

All of these used some combination of "flush" and "drain" on the serial interfaces.

"flush" and "drain" have their uses.
While the flush() call on Adruino 1.0 solves the TX "drain" issue,
(admittedly the "drain" issue on 1.0 didn't exist on pre 1.0 since the TX side was non buffered)
having a real "flush" on the RX side is also quite useful in
some situations.

In my view, once you start to use TX and RX buffers, you often need a way to manage the data in the buffers
beyond just being able to stuff a byte in the TX buffer and yank a byte from the RX buffer.
"flush" and "drain" are about helping to manage the RX and TX buffers.

--- bill

[Reply #13 on:]

mellis -

Thank you for updating the documentation to address my concerns.  Much appreciated!

I agree with others who have written here that:
1) Behavior between HardwareSerial and SoftwareSerial should be consistent.
2) Ideally, names should be consistent with behavior.
3) Until (1) and (2) are achieved, the actual behavior should be documented clearly, and perhaps with a WARNING or at least a programming tip if there is a discrepancy.

Thanks again!