I am trying to use CoolTerm (v1.9 up to v2.0.1 on Windows and MacOS) with high speed (USB) serial data input bursts (up to 2Mbps) and 20-25% of data is consistently getting dropped. The problem is much worse on MacOS where the console display also starts to lag incoming data by 10+ minutes. For comparison, Arduino Serial Monitor is able to capture and display all data without any drops.
Not sure if this is an inherent limitation but any potential fix would be appreciated.
Dropping data with high speed input
-
roger
- Site Admin
- Posts: 434
- Joined: Fri Apr 24, 2009 12:41 am
- Contact:
Re: Dropping data with high speed input
Hi,
CoolTerm uses a native textArea control that is continuously being updated with the contents of the receive data buffer. Refreshing the this control can become slow when there is a lot of data in the buffer and new data is being received at a high rate. This is unfortunately a limitation of the text control, and for 99% of all users this shouldn't be too impactful. But for some, with large receive buffers, things can get slow if the buffer starts getting full and new data comes in at a high rate. However, there shouldn't be a loss of data. Once the buffer is full, it it simply wraps around. It's a circular buffer that loses the oldest data when new data is received. I'm not sure why data is lost in your case. Does it appears if certain data is not received at all? Or is it received but then disappears?
How large did you make your receive buffer? If you don't need to scroll far back in receive data history, then I recommend keeping the buffer as small as possible. That should speed things up. In addition, I recommend enabling as few of the special character handling options as possible so that less processing of the received data is performed. Also, if you need to retain large amounts of data, then I recommend using the file capture feature to have the data written to disk as it is received. As long as you uncheck the "Leave File open while capturing", you can even inspect the capture file in real time with a "tail" capable text editor, such as the Console that comes with macOS or "Tail for Win32" (https://tailforwin32.sourceforge.net/)
I hope this helps.
Roger
PS: I'm working on an option for use cases such as yours that may result in CoolTerm getting sluggish with large amounts of data. This option essentially decouples the receive buffer from the textArea display. Instead of receiving new data into the buffer and then updating the text display with the contents of the entire buffer, any newly received data is directly appended to the text display, in addition to the receive buffer. This should speed things up significantly, but could possibly affect some special character handling features. Let me know if you're interested in testing this option.
CoolTerm uses a native textArea control that is continuously being updated with the contents of the receive data buffer. Refreshing the this control can become slow when there is a lot of data in the buffer and new data is being received at a high rate. This is unfortunately a limitation of the text control, and for 99% of all users this shouldn't be too impactful. But for some, with large receive buffers, things can get slow if the buffer starts getting full and new data comes in at a high rate. However, there shouldn't be a loss of data. Once the buffer is full, it it simply wraps around. It's a circular buffer that loses the oldest data when new data is received. I'm not sure why data is lost in your case. Does it appears if certain data is not received at all? Or is it received but then disappears?
How large did you make your receive buffer? If you don't need to scroll far back in receive data history, then I recommend keeping the buffer as small as possible. That should speed things up. In addition, I recommend enabling as few of the special character handling options as possible so that less processing of the received data is performed. Also, if you need to retain large amounts of data, then I recommend using the file capture feature to have the data written to disk as it is received. As long as you uncheck the "Leave File open while capturing", you can even inspect the capture file in real time with a "tail" capable text editor, such as the Console that comes with macOS or "Tail for Win32" (https://tailforwin32.sourceforge.net/)
I hope this helps.
Roger
PS: I'm working on an option for use cases such as yours that may result in CoolTerm getting sluggish with large amounts of data. This option essentially decouples the receive buffer from the textArea display. Instead of receiving new data into the buffer and then updating the text display with the contents of the entire buffer, any newly received data is directly appended to the text display, in addition to the receive buffer. This should speed things up significantly, but could possibly affect some special character handling features. Let me know if you're interested in testing this option.
-
mcu8484
- Posts: 9
- Joined: Tue Feb 02, 2021 5:10 pm
Re: Dropping data with high speed input
Thanks for the suggestions. I tried reducing the buffer size by 90% and capture to file but that only reduced loss to the 15% range. The display lag and loss are 2 separate but potentially related issues. It appears the internal receive buffer is overflowing with high speed data input and I suspect the display lag due to the TextArea component is a contributing factor. I think decoupling the data reception and display is critical for high speed data and I would be happy to try it when available.
-
roger
- Site Admin
- Posts: 434
- Joined: Fri Apr 24, 2009 12:41 am
- Contact:
Re: Dropping data with high speed input
Thanks for your reply.
Have you tried using a small receive buffer and instead using file capture to write the data to disk?
Roger
Have you tried using a small receive buffer and instead using file capture to write the data to disk?
Roger