roger wrote:Hi Steve,
thanks for your feedback.
steve wrote:I noticed a small difference today in the escape sequence filtering. The code snippets below show a portion of the CoolTerm window and captured text of the same CoolTerm window. CoolTerm terminal settings are set to Raw Mode and Filter ASCII Escape Sequences. The receive settings include the Plain Text capture format.
You will notice at the timestamp 001658146583 the escape sequence in the captured text is not completely filtered out.
Hmm, yeah, that's weird. The filtering code for the display and the capture SHOULD be the same. I'll take a closer look to double-check.
I checked the code, and I can explain this behavior.
As data is received by the serial port, it first gets added to the receive buffer of the operating system, where an app like CoolTerm can then go and fetch it from the buffer. The serial API is event driven, which means that CoolTerm is made aware via an event that new data is available. In addition, CoolTerm frequently polls the port to check for new data. The whole process is asynchronous, which means that there is a little bit of time that passes between data first arriving in the receive buffer and CoolTerm being aware that there is new data. On "busy" systems, this time may be longer than on idle systems. This has the effect of data arriving at CoolTerm in chunks. For high baud rates where a lot of data can arrive at the serial port in a short time, these chunks are larger than for lower baud rates.
When CoolTerm gets new data, it passes the received data to the routine that captures the data to file if capturing is enabled. It then adds the received data to its own receive databuffer whose contents are then used to update the display. Now, imagine there are ANSI escape sequences in the received data, and that the first half of one of them is right at the end of one received chunk and the remainder is at the beginning of the next chunk. The capture routine filters out all escape sequences (if enabled). However, the end of the first chunk has an incomplete escape sequence, and is therefore not recognized as such and thus left alone and the processed data is written to file. The same applies for the next chunk that begins with the remainder of the escape sequence. Again, the code doesn't recognize it as a complete sequence and writes it to file. This is how an escape sequence can end up in captured data even if filtering is enabled.
For the displayed data, this is not really a problem. After the arrival of the first chunk, the receive data buffer of CoolTerm contains an incomplete escape sequence, which will be displayed. However, immediately after, the remainder of the sequence arrives, and now the buffer has a complete sequence, which will promptly be removed when the display is refreshed again. Since such chunks arrive immediately back to back, the intermediate refresh with the incomplete escape sequence is not noticeable.
So, to avoid this from happening, Coolterm would need to wait until the whole escape sequence has arrived before processing the data and writing it to file. CoolTerm has an option for exactly this type of scenario. Since your target sends data that is terminated with some sort of "end of line", such as CR, LF, CR+LF, you can enable "Wait for Termination String" and specify what this termination is. CR+LF (OD OA in hex) is the default. HOWEVER, CoolTerm currently only waits for termination strings when timestamps are enabled. So, if you want to use this feature, you will need to turn timestamps on. I will probably make this available independent of the timestamp feature in the upcoming v2.0.0 release.