CoolTerm.py v1.8: LookAhead() / ReadAll() silently truncate large responses — _SendPacket() does a single recv()

Not sure where else to go? Then check here! This forum is for general questions and comments about my Freeware.
Post Reply
kenwoo777
Posts: 1
Joined: Sat Sep 12, 2026 2:49 am

CoolTerm.py v1.8: LookAhead() / ReadAll() silently truncate large responses — _SendPacket() does a single recv()

Post by kenwoo777 »

Hi Roger,

First of all, thank you for CoolTerm — the Data Forwarding + NULL Device
combination let me mirror a live serial session into a second window and read it
from a script without disturbing the original terminal at all. It works
beautifully; in a 60 s test at 921600 baud the mirrored copy was byte-identical
to the source (5,547,776 bytes, matching SHA-256).

While building that I ran into a bug in the bundled Python module.

ENVIRONMENT
CoolTerm 2.4.0 (2.4.0.3.0.1425), Windows 11 Pro 26100, 64-bit
Scripting/Python/CoolTerm.py v1.8, January 2025
Python 3.13.5

WHAT HAPPENS
When a window's receive buffer is large, LookAhead() returns only a prefix of
it, with no error and no indication that anything is missing. Two measurements
from the same session:

BytesAvailable() = 389,188 -> len(LookAhead()) = 61,508
BytesAvailable() = 292,653 -> len(LookAhead()) = 30,509

The same applies to ReadAll() / LookAheadHex() / GetAllParameters(), i.e. any
command with a large response.

CAUSE
_SendPacket() reads the reply with a single recv() and no read loop:

self.skt.sendall(Packet)
data = self.skt.recv(65535)
return data

TCP is free to deliver the response in several segments, so whatever has not
arrived yet at that instant is lost. _getData() then slices Packet[6:6+LEN]
out of the short buffer and returns a silently shortened string. The two
numbers above are not round, and differ between calls, which is consistent
with partial reads rather than a fixed cap.

HOW TO REPRODUCE
1. Let a window accumulate more than ~64 KB in its receive buffer
(RXBufferSize set high, or feed it with Receive()).
2. Compare BytesAvailable(ID) with len(LookAhead(ID)).

SUGGESTED FIX
Read the 6-byte header first, then exactly LEN payload bytes:

Code: Select all

      def _recv_exact(self, n):
          buf = b""
          while len(buf) < n:
              chunk = self.skt.recv(n - len(buf))
              if not chunk:
                  raise ConnectionError("socket closed by CoolTerm")
              buf += chunk
          return buf

      def _SendPacket(self, Packet):
          self.skt.sendall(Packet)
          head = self._recv_exact(6)
          LEN = int.from_bytes(head[1:3], byteorder="little")
          return head + self._recv_exact(LEN)
With this change every response comes back complete in my tests.

QUESTION ABOUT THE PROTOCOL
The length field in the packet header is 2 bytes, so a single response can
carry at most 65,535 bytes. What does CoolTerm do when the requested data is
larger than that — is the response capped at 65,535, is the data truncated on
your side, or is there a continuation mechanism I have missed? Knowing this
would tell client authors whether they must keep RXBufferSize below 64 KB, or
drain the buffer with Read() in chunks, to be safe. It may be worth a note in
the protocol PDF either way.

TWO SMALLER OBSERVATIONS
1. The remote control socket appears to accept only one client connection at a
time — a second CoolTermSocket() prints "ERROR: Could not connect to
CoolTerm" while the first is still open. Is that by design? It is easy to
work around once you know, but it is not mentioned in the help.
2. Receive(ID, Data) returns True when the target window's port is closed, but
the data does not appear in the receive buffer (BytesAvailable stays 0).
Returning False, or documenting that the port must be open, would make this
less surprising.

Thanks again for the tool, and for keeping the protocol documented — being able
to fix the client myself is exactly why that matters.

--
Ken, with Claude (Anthropic's AI assistant) as investigation partner.
Every number above comes from an actual run against real hardware on the machine
described, not from reading the code alone — happy to re-run anything or test a
patched CoolTerm.py if that would help.
User avatar
roger
Site Admin
Posts: 573
Joined: Fri Apr 24, 2009 12:41 am
Contact:

Re: CoolTerm.py v1.8: LookAhead() / ReadAll() silently truncate large responses — _SendPacket() does a single recv()

Post by roger »

kenwoo777 wrote: Sat Sep 12, 2026 3:05 am Hi Roger,

First of all, thank you for CoolTerm — the Data Forwarding + NULL Device
combination let me mirror a live serial session into a second window and read it
from a script without disturbing the original terminal at all. It works
beautifully; in a 60 s test at 921600 baud the mirrored copy was byte-identical
to the source (5,547,776 bytes, matching SHA-256).
Great! I'm glad to hear it. And I'm glad to hear that data forwarding and the NULL device are used in the field. These are some of the features that most casual users will probably never user.
While building that I ran into a bug in the bundled Python module.
Uh oh :(
ENVIRONMENT
CoolTerm 2.4.0 (2.4.0.3.0.1425), Windows 11 Pro 26100, 64-bit
Scripting/Python/CoolTerm.py v1.8, January 2025
Python 3.13.5

WHAT HAPPENS
When a window's receive buffer is large, LookAhead() returns only a prefix of
it, with no error and no indication that anything is missing. Two measurements
from the same session:

BytesAvailable() = 389,188 -> len(LookAhead()) = 61,508
BytesAvailable() = 292,653 -> len(LookAhead()) = 30,509

The same applies to ReadAll() / LookAheadHex() / GetAllParameters(), i.e. any
command with a large response.

CAUSE
_SendPacket() reads the reply with a single recv() and no read loop:

self.skt.sendall(Packet)
data = self.skt.recv(65535)
return data

TCP is free to deliver the response in several segments, so whatever has not
arrived yet at that instant is lost. _getData() then slices Packet[6:6+LEN]
out of the short buffer and returns a silently shortened string. The two
numbers above are not round, and differ between calls, which is consistent
with partial reads rather than a fixed cap.

HOW TO REPRODUCE
1. Let a window accumulate more than ~64 KB in its receive buffer
(RXBufferSize set high, or feed it with Receive()).
2. Compare BytesAvailable(ID) with len(LookAhead(ID)).

SUGGESTED FIX
Read the 6-byte header first, then exactly LEN payload bytes:

Code: Select all

      def _recv_exact(self, n):
          buf = b""
          while len(buf) < n:
              chunk = self.skt.recv(n - len(buf))
              if not chunk:
                  raise ConnectionError("socket closed by CoolTerm")
              buf += chunk
          return buf

      def _SendPacket(self, Packet):
          self.skt.sendall(Packet)
          head = self._recv_exact(6)
          LEN = int.from_bytes(head[1:3], byteorder="little")
          return head + self._recv_exact(LEN)
With this change every response comes back complete in my tests.

QUESTION ABOUT THE PROTOCOL
The length field in the packet header is 2 bytes, so a single response can
carry at most 65,535 bytes. What does CoolTerm do when the requested data is
larger than that — is the response capped at 65,535, is the data truncated on
your side, or is there a continuation mechanism I have missed? Knowing this
would tell client authors whether they must keep RXBufferSize below 64 KB, or
drain the buffer with Read() in chunks, to be safe. It may be worth a note in
the protocol PDF either way.
It seems you have come across a limitation in the protocol in didn't anticipate to ever be a problem when I first came up with this scheme. A shortsightedness on my part along the lines of what Bill Gates said about the amount RAM anybody might ever need: "640K ought to be enough for anybody". I guess my assumption was "why would anybody in their right mind ever want to send back and forth more than 64kB of data?" :-)
communicating at nearly 1MB/s where 64kB of data can be received in less than 1s wasn't really a use case that was on my mind.

When the RemoteControl Socket in CoolTerm assembles a data packet, it calculates the length of the data payload and stores it in a UInt16 variable. Thus, for any payload larger than 65535 bytes, this value rolls over. That is the LEN value of the packet that is sent. Furthermore, the socket code doesn't currently care what the length of the payload is. It simply adds it to the packet. So, when there are 389,188 in the buffer, and the client does a LookAhead, the packet will contain all 389,188 bytes, but the LEN field value is 389,188 MOD 65536 = 61,508, because the LEN value rolled over several times and now holds the remainder. The socket in CoolTerm.py should actually receive ALL the data in the CoolTerm receive buffer (I haven't tested this, but from looking at the code it appears that's what it should do), but the call to LookAhead() in CoolTerm.py only returns the first LEN bytes (that happens in _getData()). This explains the discrepancy you observed.

This is incorrect behavior and needs to be fixed somehow. In the least, the documentation should reflect that the receive buffer should be kept at max 65536 bytes to avoid issues.
- OP_READ with an requested length of more than 65536 should return an ACK_BAD_ARGUMENT packet instead of a data.
- OP_READ_HEX with an requested length of more than 32678 should return an ACK_BAD_ARGUMENT packet instead of a data.
- OP_LOOK_AHEAD and OP_READ_ALL should cap the payload to the first 65536 bytes in the receive buffer when there is more data than that.
- OP_READ_ALL_HEX should cap the payload to the first 32768 bytes in the receive buffer when there is more data than that.
- Similarly, CoolTerm.py should be updated to include some sanity checking and throw exceptions when a user attempts to transfer more than what a data packet can hold as a payload.

You mentioned GetAllParameters() above. Have you had any issue with that command? That payload should be much smaller than 64kB.

Which script command did you use with your suggested fix above? LookAhead()? While not having actually tested it, it extracts a potentially incorrect LEN value (when there is more than 65536 bytes of data in the payload) from the response, and I'm not quite sure how it fixes this issue as _getData(), which gets called on the response returned by _sendPacket() returns the first LEN bytes from the response, which should result in the same as your fix. Perhaps I'm missing something. It does prevent making the receive buffer of the socket unnecessary long, though, so I will probably end up updating my code with a version of it.
I do know that the line data = self.skt.recv(65535) in _sendPacket() in the current code could be an issue should be changed to data = self.skt.recv(6+65536) so that no data is lost when the payload is actually 65636 bytes long. But that does specify a receive buffer size that is not always needed. So, your fix to read the LEN field and then set the receive buffer size accordingly makes sense.

TWO SMALLER OBSERVATIONS
1. The remote control socket appears to accept only one client connection at a
time — a second CoolTermSocket() prints "ERROR: Could not connect to
CoolTerm" while the first is still open. Is that by design? It is easy to
work around once you know, but it is not mentioned in the help.
2. Receive(ID, Data) returns True when the target window's port is closed, but
the data does not appear in the receive buffer (BytesAvailable stays 0).
Returning False, or documenting that the port must be open, would make this
less surprising.
1. Yes, that is by design. The TCP socket I'm using can only accept one client connection at a time. While I could use a socket that can accept multiple connections by handing off existing connections to another port and free up the listening port for new connections, I did this specifically to avoid weird issue can can arise when multiple clients are connected, such as 2 simultaneous connections sending OP_READ and both emptying out the receive buffer, etc.

2. Yes, I should probably document this. It's always a good idea to check IsConnected() before trying to exchange any data over the serial port.
Thanks again for the tool, and for keeping the protocol documented — being able
to fix the client myself is exactly why that matters.

--
Ken, with Claude (Anthropic's AI assistant) as investigation partner.
Every number above comes from an actual run against real hardware on the machine
described, not from reading the code alone — happy to re-run anything or test a
patched CoolTerm.py if that would help.
I'm glad you find it useful. And thank you very much for your feedback and your investigation effort. It's is truly appreciated. It revealed a limitation that I didn't properly anticipate.


PS: I'm working on a next major release of CoolTerm. Perhaps it's time to revamp the protocol to handle larger payloads. But I will have to do it in a way that won't break any scripting or automation anybody has already implemented. Perhaps I will add new OP-code to the protocol that let's the user switch to a new packet format with a 4-byte LEN field if they need it and update the CoolTerm.py accordingly. "4GB of payload ought to be enough for anybody", right? :-)
Post Reply