Slight difference in escape sequence filtering

[steve]

I noticed a small difference today in the escape sequence filtering. The code snippets below show a portion of the CoolTerm window and captured text of the same CoolTerm window. CoolTerm terminal settings are set to Raw Mode and Filter ASCII Escape Sequences. The receive settings include the Plain Text capture format.

You will notice at the timestamp 001658146583 the escape sequence in the captured text is not completely filtered out.

Code: Select all

001658146223,s2=-9,s3=0
001658146333,s0=3,s1=2
001658146343,s2=-8,s3=0
001658146453,s0=3,s1=2
001658146463,s2=-8,s3=0
001658146483,t=2260,h=5424,p=9338
001658146573,s0=2,s1=2
001658146583,s2=-8,s3=0
001658146693,s0=3,s1=2
001658146703,s2=-8,s3=0
001658146783,t=2264,h=5431,p=9339

Code: Select all

001658146223,s2=-9,s3=0
001658146333,s0=3,s1=2
001658146343,s2=-8,s3=0
001658146453,s0=3,s1=2
001658146463,s2=-8,s3=0
001658146483,t=2260,h=5424,p=9338
001658146573,s0=2,s1=2
001.[1;36m658146583,s2=-8,s3=0
001658146693,s0=3,s1=2
001658146703,s2=-8,s3=0
001658146783,t=2264,h=5431,p=9339

I've noticed some other odd behaviour. For instance, the prompt appears to be duplicated in this snippet (escape filtering turned on):

Code: Select all

uart:~$ uart:~$ [00:40:21.029,846] <wrn> app: network unreachable
uart:~$ uart:~$ [00:40:41.132,965] <wrn> app: network unreachable
uart:~$ uart:~$ [00:40:51.133,605] <wrn> app: network unreachable
uart:~$ uart:~$ [00:42:41.133,605] <wrn> app: network unreachable
uart:~$ uart:~$ [00:42:51.132,934] <wrn> app: network unreachable
uart:~$ uart:~$

The "screen" utility displays the information correctly:

Code: Select all

[13:40:38.817,535] <wrn> app: network unreachable
[13:40:48.817,504] <wrn> app: network unreachable
[13:42:38.817,504] <wrn> app: network unreachable
[13:42:48.816,680] <wrn> app: network unreachable
uart:~$

It would be great if CoolTerm could implement the same emulation that screen provides.

[steve]

A couple things. With respect to the duplicated prompt, I should mention that if you look closely at the text, the actual prompt is ok. That is, if I press the return key and will see a single prompt. It is the lines that have the log messages that have the double prompt where there shouldn't be a prompt displayed at all (the first thing should be the timestamp).

I also noticed that when escape filter is turned on, backspace does not work. That is, when you press the backspace key (macOS "delete"), the cursor does not move backward and erase the previous character. However, if a log message arrives while typing the command line, the command line is re-written with the backspaced text removed. I attached a snippet of text that illustrates this issue below. The second lines shows the text "this is some text". I then backspaced five characters and waited for the log message to arrive. When the log message arrived, the line was rewritten without the characters " text".

Code: Select all

uart:~$ uart:~$ [00:56:51.132,965] <wrn> app: network unreachable
uart:~$ uart:~$ this is some text[00:58:41.132,995] <wrn> app: network unreachable
uart:~$ this is someuart:~$ this is some[00:58:51.132,629] <wrn> app: network unreachable
uart:~$ this is someuart:~$ this is some

I think this is likely an unfortunate consequence of "filtering" the escape sequences rather than implementing the escape sequences.

[roger]

Hi Steve,

thanks for your feedback.

I noticed a small difference today in the escape sequence filtering. The code snippets below show a portion of the CoolTerm window and captured text of the same CoolTerm window. CoolTerm terminal settings are set to Raw Mode and Filter ASCII Escape Sequences. The receive settings include the Plain Text capture format.

You will notice at the timestamp 001658146583 the escape sequence in the captured text is not completely filtered out.

Hmm, yeah, that's weird. The filtering code for the display and the capture SHOULD be the same. I'll take a closer look to double-check.

I've noticed some other odd behaviour. For instance, the prompt appears to be duplicated in this snippet (escape filtering turned on):

The "screen" utility displays the information correctly:

It would be great if CoolTerm could implement the same emulation that screen provides.

I assume you're seeing the double prompt on each line in the CoolTerm window? If so, can you reproduce this and then switch over to Hex view to see the raw data that is being received? If the plain text view shows all these additional prompts, then it must be because they were actually received. Perhaps your target sends an escape sequence to delete the extra prompts.

I have added some rudimentary handling of SOME escape sequences (such as e.g. converting ESC[#D sequences to backspaces) to the upcoming v2.0.0 release. I'd like to encourage you to give the latest beta a try. I added you to the Beta group in the forums. You should now see a "Betas" channel under the "Freeware" header. The very first topic at the top of the Betas channels has the current download link for the latest betas.
Alternatively, you can also join the betas mailing list at http://the-meiers.org/mailman/listinfo/ ... meiers.org to keep up to date.

[roger]

A couple things. With respect to the duplicated prompt, I should mention that if you look closely at the text, the actual prompt is ok. That is, if I press the return key and will see a single prompt. It is the lines that have the log messages that have the double prompt where there shouldn't be a prompt displayed at all (the first thing should be the timestamp).

I also noticed that when escape filter is turned on, backspace does not work. That is, when you press the backspace key (macOS "delete"), the cursor does not move backward and erase the previous character. However, if a log message arrives while typing the command line, the command line is re-written with the backspaced text removed. I attached a snippet of text that illustrates this issue below. The second lines shows the text "this is some text". I then backspaced five characters and waited for the log message to arrive. When the log message arrived, the line was rewritten without the characters " text".

I think this is likely an unfortunate consequence of "filtering" the escape sequences rather than implementing the escape sequences.

The fact that backspace doesn't work when escape filtering is turned on seems to indicate that your target is in fact sending back an escape sequence to delete a character rather than and the ASCII code for backspace or delete. CoolTerm can handle BS and DELETE characters, but this has to be enabled under special character handling. However, this won't help when your target sends an escape sequence to delete characters. But the latest beta might work because it can convert ESC[#D sequences to backspaces, which CoolTerm can then handle properly. See the previous post for information on how to get you hands on the latest beta for testing.

[roger]

Hi Steve,

thanks for your feedback.

I noticed a small difference today in the escape sequence filtering. The code snippets below show a portion of the CoolTerm window and captured text of the same CoolTerm window. CoolTerm terminal settings are set to Raw Mode and Filter ASCII Escape Sequences. The receive settings include the Plain Text capture format.

You will notice at the timestamp 001658146583 the escape sequence in the captured text is not completely filtered out.

Hmm, yeah, that's weird. The filtering code for the display and the capture SHOULD be the same. I'll take a closer look to double-check.

I checked the code, and I can explain this behavior.
As data is received by the serial port, it first gets added to the receive buffer of the operating system, where an app like CoolTerm can then go and fetch it from the buffer. The serial API is event driven, which means that CoolTerm is made aware via an event that new data is available. In addition, CoolTerm frequently polls the port to check for new data. The whole process is asynchronous, which means that there is a little bit of time that passes between data first arriving in the receive buffer and CoolTerm being aware that there is new data. On "busy" systems, this time may be longer than on idle systems. This has the effect of data arriving at CoolTerm in chunks. For high baud rates where a lot of data can arrive at the serial port in a short time, these chunks are larger than for lower baud rates.

When CoolTerm gets new data, it passes the received data to the routine that captures the data to file if capturing is enabled. It then adds the received data to its own receive databuffer whose contents are then used to update the display. Now, imagine there are ANSI escape sequences in the received data, and that the first half of one of them is right at the end of one received chunk and the remainder is at the beginning of the next chunk. The capture routine filters out all escape sequences (if enabled). However, the end of the first chunk has an incomplete escape sequence, and is therefore not recognized as such and thus left alone and the processed data is written to file. The same applies for the next chunk that begins with the remainder of the escape sequence. Again, the code doesn't recognize it as a complete sequence and writes it to file. This is how an escape sequence can end up in captured data even if filtering is enabled.

For the displayed data, this is not really a problem. After the arrival of the first chunk, the receive data buffer of CoolTerm contains an incomplete escape sequence, which will be displayed. However, immediately after, the remainder of the sequence arrives, and now the buffer has a complete sequence, which will promptly be removed when the display is refreshed again. Since such chunks arrive immediately back to back, the intermediate refresh with the incomplete escape sequence is not noticeable.

So, to avoid this from happening, Coolterm would need to wait until the whole escape sequence has arrived before processing the data and writing it to file. CoolTerm has an option for exactly this type of scenario. Since your target sends data that is terminated with some sort of "end of line", such as CR, LF, CR+LF, you can enable "Wait for Termination String" and specify what this termination is. CR+LF (OD OA in hex) is the default. HOWEVER, CoolTerm currently only waits for termination strings when timestamps are enabled. So, if you want to use this feature, you will need to turn timestamps on. I will probably make this available independent of the timestamp feature in the upcoming v2.0.0 release.