Support for STX

[Kaiserdragon2]

I have tried to run the script with the Cycplus M2 after some modifications.
The problem i have now is that sometimes the M2 sends 128 byte data and sometimes 1024 byte data and i don't know how to modify the script to handle both cases. It would be nice to have this implemented or could you give me some tips on how to do this .

[ekspla]

I am not quite sure if the following idea works/is helpful to you because
I do not have a real device that uses STX to test.

First of all, we should increase self.block_buf size from 128+5=133 to
1024+5=1029 bytes.

self.block_data and self.block_crc in def __init__(self) in the
original version do not work. We have to implement a mechanism to check
if the block is SOH/STX at the beginning of check_block_buf().
Something like:

if self.idx_block_buf > 0 and self.block_buf[0] == 1: # SOH
    self.block_is_stx = False
elif self.idx_block_buf > 0 and self.block_buf[0] == 2: # STX
    self.block_is_stx = True

and await for the buffer to be filled (133/1029 bytes). When the buffer is
filled, we can take self.block_data and self.block_crc using memoryview.

if self.block_is_stx:
    self.block_data = self.mv_block_buf[3:-2]
    self.block_crc = self.mv_block_buf[-2:]
else:
    self.block_data = self.mv_block_buf[3:131]
    self.block_crc = self.mv_block_buf[131:133]

Alternatively, we may do all of these tasks in notification_handler().

I hope your success in making a working code for your M2.

[Kaiserdragon2]

Thanks for your help this works prefectly.
Just one other Question i needed to change VALUE_IDLE to "0x04" instead of "0x04 0x00 0x04" for the M2 and the site you mentioned in your readme, that describes the YModem protocol, has 04 for Idle in the table for IDLE in section 5.3.2 and in the graphic in section 5.3.3 it is 04 00 04. Do you know what this is about ?

[ekspla]

All of the commands/responses should be followed by a data and a crc8_xor as described in the official document.
e.g. A command to fetch filelist.txt is [0x05 (command), 'filelist.txt', 0x57 (crc8_xor)].

If no data, any length (longer or equal to one) of 0x00 should be used as the data. Please see def make_command()
in my original version.

So a bytearray of [0x04 (command), 0x00 (dummy data), 0x04 (crc8_xor)] (VALUE_IDLE in my code) is used for IDLE
command/response as is. We should not use make_command() in this case because the bytearray is the output of
make_command(bytearray([0x04, ])). This is also the case in using VALUE_DISKSPACE in my code.

A question from me (for curiosity): does your M2 device supports data length extension (payloads greater than 20-byte length, bluetooth 4.2) and/or 2M PHY (Bluetooth 5.0)?

[ekspla]

A branch for additional support of STX in YMODEM for CPython version was created, though not tested with a real device with STX. This code was tested with XOSS-G+ (SOH) on Linux/Win10/Win11.

[Kaiserdragon2]

A question from me (for curiosity): does your M2 device supports data length extension (payloads greater than 20-byte length, bluetooth 4.2) and/or 2M PHY (Bluetooth 5.0)?

2M PHY isn't supported
MTU is 185 (does this answer the payload size ? or how can i check it?)

The branch with STX support works for the M2 after changing value idle to VALUE_IDLE = bytearray([0x04])

[ekspla]

A question from me (for curiosity): does your M2 device supports data length extension (payloads greater than 20-byte length, bluetooth 4.2) and/or 2M PHY (Bluetooth 5.0)?
2M PHY isn't supported
MTU is 185 (does this answer the payload size ? or how can i check it?)

MTU is in higher level of bluetooth (ATT), while DLE is in the lower.
Even if DLE is not enabled, an MTU larger than the payload can be used
by fragmentation.

Thinking the other way around, MTU should be increased to make good
use of DLE. Sophisticated bluetooth stacks/apps do this automatically.

So, the higher MTUs may be a sign that DLE is enabled (but is not
necessarily true).

So, a 1029-byte STX block is fragmented by the MTU (185), and then
may, or may not, be fragmented again by the payload.

The branch with STX support works for the M2 after changing value idle to VALUE_IDLE = bytearray([0x04])

I am glad that the branch works as expected, while another question
remains open:
why do we have to use [0x04, ] instead of [0x04, 0x00, 0x04] with
your M2?

[Kaiserdragon2]

After looking at some sniffed Bluetooth communications, I got a 040004 response after a file transfer.
It looked like this:

M2                                         PC
   -----------filesdata---------------->>>               
   <<<------------06---------------------
   ---------------04------------------->>>
   <<<------------15---------------------
   ---------------04------------------->>>
   <<<------------06---------------------
   -------------040004---------------->>>
   <<<----------ff00ff--------------------
   ---------------04------------------->>>
M2                                         PC

[ekspla]

@Kaiserdragon2 Thanks for the update.

From unknown reason, which is not in the official document, it seems that your M2 sends
an additional 0x04 after the IDLE response while this is not the case in my G+.

Can you try a fix like this?

    async def end_of_transfer(self, client):
        # The first EOT was received already.
        await asyncio.sleep(0.1) # This avoids NAK to be sent too fast.
        self.notification_data = AWAIT_NEW_DATA
        await self.send_cmd(client, RX_CHARACTERISTIC_UUID, VALUE_NAK, 0.1) # Send NAK.
        await self.wait_until_data(client)                                   # Receive the second EOT.
        await asyncio.sleep(0.1) # This avoids ACK to be sent too fast.
        self.notification_data = AWAIT_NEW_DATA
        await self.send_cmd(client, RX_CHARACTERISTIC_UUID, VALUE_ACK, 0.1) # Send ACK.
        await self.wait_until_data(client)                                   # Receive IDLE (0x04, 0x00, 0x04)
+       if self.notification_data == VALUE_IDLE:
+           await self.wait_until_data(client)
+           self.notification_data = VALUE_IDLE

The additional wait_until_data() should timeout in 10 seconds if the third EOT were not sent. You can
reduce the timeout in wait_until_data() if you want.

Best wishes

[ekspla]

@Kaiserdragon2 An additional comment:

The last comment I wrote was based on an assumption that you took the log by using
HCI sniffing tool. i.e. the 0x04 sent before the [0xff, 0x00, 0xff] was captured
back to front.

If this were not the case (using BLE sniffer in correct time order) we would have to
modify the code in get_idle_status() and this would result in incompatible protocol
to XOSS.

The latter seems unlikely to me at the moment.

[Kaiserdragon2]

The fix did not work ( with VALUE_IDLE = bytearray([0x04, 0x00, 0x04])).
I used an nrf52840 dongle in combination with Wireshark to capture the traffic.
The capture was between the M2 and the modified script. I need to redo this with the original app investigate this a bit further.

[ekspla]

The capture was between the M2 and the modified script. I need to redo this with the original app investigate this a bit further.

I continue to welcome your inputs.

[Kaiserdragon2]

I have captured the traffic with the original app and it looks the same. The idle response for the M2 seems to be 04 instead of 040004. I have append the pcap file for further investigation. The way it looks to me the protocol isn`t 100% the same and the Xoss app as a way to differentiate the devices to work with both.
M2XossAppPCAP.zip

[ekspla]

@Kaiserdragon2 Thanks for the update.

From the pcap of Cycplus M2 I saw that:

1. LE Data length extension (DLE) was used (185 octets),

2. no response to the 'T' (time) command (similar to my XOSS G+ gen1),

3. [0x09, 0x00, 0x09] was used to obtain disk-free while a longer command of
   0900000000000000000000000000000000000009s (also similar form used in 4)
   were used in my XOSS app,

4. IDLE response to the [0xff, 0x00, 0xff] command in M2 was a single 0x04 instead
   of [0x04, 0x00, 0x04] while this was not the case after the EOTs ([0x04, 0x00, 0x04]
   was sent in line with the protocol, see No.258-260), and

5. the zeroth block in YMODEM was SOH (128) while the following blocks fragmented
   via L2CAP were STX (1024 bytes).

From 4), I have a feeling that the single 0x04 used for IDLE response is a bug in M2's
firmware and XOSS app that might be fixed in their future updates; your XOSS app itself
seems to be checking only the first byte in the response. This is because the correct
responses in line with the XOSS protocol were used after the EOTs.

Also, please see the attached my pcap of XOSS G+ and the XOSS app which shows the
[0x04, 0x00, 0x04] response.
XOSS_Gplus_App_extracted.zip

So a better fix for your M2 for the moment would be to modify as following:

    async def get_idle_status(self, client):
+       def fix_idle():
+           if (len(self.notification_data) == 1 and 
+               self.notification_data[0] == VALUE_IDLE[0]):
+               self.notification_data = VALUE_IDLE
        self.notification_data = AWAIT_NEW_DATA
        self.is_download = self.is_upload = False
        await self.send_cmd(client, CTL_CHARACTERISTIC_UUID, VALUE_STATUS, 5.0)  # Send STATUS (0xff, 0x00, 0xff)
        await self.wait_until_data(client)
+       fix_idle()
        if self.notification_data == VALUE_IDLE:                                  # Receive IDLE (0x04, 0x00, 0x04)
            return True
        if self.notification_data == AWAIT_NEW_DATA:                              # Timeout; No response
            await self.send_cmd(client, CTL_CHARACTERISTIC_UUID, VALUE_IDLE, 0.1) # Send IDLE (0x04, 0x00, 0x04)
            await self.wait_until_data(client)
+           fix_idle()
            if self.notification_data == VALUE_IDLE:                              # Receive IDLE (0x04, 0x00, 0x04)
                return True
        print(f'Error: {self.notification_data}')
        return False

[Kaiserdragon2]

The proposed workaround works great. I'm not sure if the firmware of the M2 ever will receive a fix it hasn't been updated in over a year.

[ekspla]

I am glad that the workaround for your Cycplus M2 was useful.

Now that STX is supported by the PRs (#2 and #5) and
another issue, possibly a bug in the firmware, related to IDLE
in Cycplus M2 was solved by the workaround, let's mark this
issue as closed.

Thank you for your contributions. Feel free to open a new issue
if you find something.