Coolterm hangs - receive buffer too small ? (macos)

[roger]

Shouldn't it continue printing dots, starting on a new empty line after reaching the end of the line, scrolling the screen up one line? There's also no update if I interrupt the procedure and execute it again. The screen remains the same with the cursor sitting in the middle of the screen.

Yes, that is actually correct behavior. For every dot that is added at the end of what's on the screen, another one is removed at the beginning of the screen. So you're always seeing exactly 1000 dots when the buffer is set to 1000 bytes. The buffer is a circular buffer. As long as it receives only dots, all you'll ever see on screen is 1000 dots. It's a different set of dots each time a new one arrives, but since they are indistinguishable, the display always looks the same until something else arrives.

I have to send something else than a dot for the cursor to advance, filling the remaining space of the line, and new lines. It's like the program concludes you have received xxx number of the same character (reaching the limit of the buffer) and then decides to ignore the same character, unless you give it something else in-between.

Alternatively, as you wrote, pressing the Clear Data button revives the output.

It's not ignoring the same character. It does actually prints it at the end, but at the same time it also deletes one at the beginning. This is why the display always looks the same. If you change your script to send 500 dots, then some other character, and then another 500 dots, and repeat that, you will see this other character ripple through the 1000 dots you see on the screen.
Another thing your can try with your script to have it continuously send random characters. Once the buffer is full, the cursor will not move anymore, but the display contents are changing with each new character that is received. You will always see the last 1000 received characters.

However, this does not explain the issue I initially described with the Realtek prompt not reappearing. Could it be that the RTKSPI utility expects a specific number of characters in the output (like dots) before it decides that the operation is complete and returns control back to the user?

Correct, it doesn't. And simply increasing the size of the receive buffer wouldn't change the interaction of the RTKSPI utility. It has no idea of what's on the other side.

[roger]

I'm glad that fixes it.

macOS and Linux actually treat CR characters as a new line character in almost all situations. For example, a CR character in a text file will cause text to continue on a new line. So, the way CoolTerm is handling 0D 0D 0A is actually kind of default behavior. the first 0D is treated as a new line, and then the 0D 0A is treated as another one, hence the extra empty line.
0D, or "Carriage Return" come from the old days of type writers and teletypes, when a CR merely returned the carriage to the beginning of the line, and only when an LF was received would it advance to the next line. If, for example, a teletype received some text, followed by a CR, and then following by more text, then new text would be typed over the text that was received before the CR.
It seems that RTKSPI expects the serial terminal to behave like an old teletype. Otherwise the extra CR makes no sense as it serves no purpose.

[maxterm]

I did more tests...

When the Receive Buffer is set to 1000, then I won't see more than 1000 dots on the terminal screen, regardless. I guess I understand now what you meant earlier, saying the screen only appears to be frozen, since adding more dots won't advance the cursor, since the screen already reached the display or buffer limit of 1000 dots. This can also be confirmed when continuing to printing something else than dots. The cursor remains at position 1000, regardless of the character, while shifting the dots backwards.

I think this is the confusing part: It's the total amount of 1000 characters that sets the limit of the current cursor position on screen. Once I start to output text that with newlines, the cursor will move further down the screen and advance, but still limiting the total amount of characters on screen to 1000.

So I think I need to change my previous assessment. Scrolling appears to work as expected. You probably wouldn't want to limit the Receive Buffer to 1000 for anything else than troubleshooting, though.

But again, this does not explaining the problem I ran into writing the bios chip with the RTKSPI utility.

[maxterm]

Sorry, I didn't notice that you already replied, which makes my last response somewhat redundant, since it pretty much confirms what you explained. It requires a bit more thinking, I guess, to grasp what limiting the Receive Buffer to 1000 character actually does in practice, which is not a setting you would normally apply.

I know I mentioned line-endings, but considering, it think it will be best to address this in another topic. It's complex enough. It also just occurred to me, that I may have mixed up the tests setting the buffer to 1000, 10000, or 100000. I'll do another test.

[maxterm]

I tested the behavior of the terminal with the Receive Buffer set to both 1000 and 10000.

Without resizing the dimensions of the terminal window, it can display a total of 2632 characters, arranged as 28 rows of 94 characters each (28 x 94 = 2632).

When the Receive Buffer is set to 1000, it fills about half the terminal screen and then stops. As it reaches 1000 characters, it continues to add new characters but removes the oldest ones from the top of the screen, preventing scrolling.

In contrast, when set to 10000, the buffer fills the entire screen and scrolls properly as it approaches the terminal's limit of 2632 characters. Upon reaching this limit, it introduces a new blank line by shifting the content up into the scroll back buffer. However, once it reaches 10000 characters, the screen stops scrolling, and the cursor becomes stuck somewhere in the middle of the last line.

This behavior is not what I would expect based on my experience with other terminal applications. When the Receive Buffer is set to 10000, it should always fill the complete terminal screen without stopping in the middle of the last line. It should continue to scroll, adding a new line upon reaching the end of the screen.

I'm planning on refreshing the firmware again and can give CoolTerm another try. Is there something you would like me to troubleshoot when the terminal output gets stuck, refusing to show the Realtek prompt?

[roger]

I'm planning on refreshing the firmware again and can give CoolTerm another try. Is there something you would like me to troubleshoot when the terminal output gets stuck, refusing to show the Realtek prompt?

Thanks! Can you turn on a file capture in CoolTerm before your next attempt to refresh the firmware? That way there will be a record of what was received, even if somehow the text display gets stuck.

[maxterm]

Hi Roger,

I hope you're doing well. OK, I have a better picture now what's happening with the RTKSPI utility when the Receive Buffer is set to 10000 (default).

When entering RTKSPI WRITE at the Realtek prompt, it will generate about 16385 dots, about 80 dots every 2 seconds. The cursor moves as expected until it reaches the end of the screen, at which point the screen begins to scroll line by line. Everything so far looks normal.

However, upon reaching the buffer limit of 10,000 characters, the cursor stops mid-line. The Realtek prompt eventually appears, indicating that the process has completed. As previously noted, when the buffer limit is reached, new characters are inserted at the current cursor position, pushing out characters from the beginning of the buffer. Since the RTKSPI utility generates only dots without any newlines (CRLF), the screen does not scroll. This creates the misleading impression that the process has stalled.

I suppose past experiences were influenced by the time and byte counters in the status bar, which reflect the entire session history rather than just the RTKSPI tool's output, obviously. Hence, seeing 20 minutes and 30,000 bytes counted led to confusion, especially considering the potential risk of having just bricking the device, and no understanding what happened.

It seems that this situation arises from a combination of the RTKSPI utility’s output method and the limitations of the buffer. There's probably nothing that can be fixed, other than increasing the Receive Buffer and hoping that no tool like RTKSPI challenges the buffer limit.

What is the rationale behind limiting the Receive Buffer to 10,000 characters anyway?

I have the file capture log. I can send it you if you like, please let know.

Kind regards.

[roger]

Ah, great! I'm glad that firmware upload works.

Many, if not most, serial terminal might not have an intermediate receive buffer like CoolTerm has. The reason CoolTerm has such a thing is to enable the different display modes, like hex view and chart view. These other views are not updated when new data arrives unless they have been selected. Therefore, when switching between display modes, CoolTerm needs a local history of received data it can use to render the display in any of the other modes and display previously received data. Therefore, CoolTerm is designed such that the currently active display is always a live and accurate reflection of the receive buffer, i.e. they are always in sync. For example, when CoolTerm is in plain text view, its entire text display is updated every time a new byte arrives. This might not seem to be the most efficient way, but it's the best way to ensure the plain text display is always accurate, especially when certain formatting functions are enabled. So, to keep the CPU load minimal, it is recommended that the receive buffer is only kept as large as needed to keep CoolTerm snappy. If you have a very large receive buffer, and its (near) full, a lot of data processing needs to happen with every received byte. This can make CoolTerm sluggish, especially on older systems, and when data arrives in very frequent intervals. 10k is a somewhat arbitrary number, but on that proved more than enough for most use cases. And if retention of much more data is required, for example in data logging use cases, file capture is the best way to accomplish that.

In your case, where a limited receive buffer can lead to confusion, there is actually something that can be done without increasing the size of the receive buffer. There is a way to emulate how other terminals work by directly dumping received data into the text display without worrying about it being in sync with the receive buffer contents. In the "Receive" section of the settings, there is a "Direct RX update of text display" option. If you turn that on, the display is not updated with the receive buffer contents, but instead every new byte is directly added to the text display. CoolTerm will still update the receive buffer as before (to make it possible to switch to other modes and display previously received data), but the text display and the receive buffer are decoupled from each other. Thus, the text display can hold more text than the receive buffer. I don't know why this didn't occur to me earlier, but turning on the "Direct RX update of text display" option should have been my first recommendation as this is probably the best configuration for your particular use case.

I don't think I need the capture log since you did get the Realtek prompt after the firmware upload completed and everything seems to be working.

Cheers,

Roger

[maxterm]

I can confirm that enabling "Direct RX update" in the Receive Options solves the problem, but then the backspace/delete key no longer works.

[maxterm]

Btw, I think the display modes, like hex view and chart view, are interesting features that make CoolTerm stand out from similar tools. While these modes can be helpful for debugging terminal content, I personally wouldn't find them necessary unless I need to troubleshoot the terminal application itself.

If such features come at the cost of slowing down performance, increasing CPU load, or causing the cursor to stop mid-line under certain conditions (like reaching the Receive Buffer limit), I would reconsider. Since your software already includes a file capture feature, perhaps you could consider implementing a file viewer or converter to allow users to convert captured data into these alternative display modes instead, and have a more reliable terminal.