Use a precompiled Portduino version of firmware for python integration testing?

[geeksville]

Hi @ianmcorvidae and @thebentern,

I'm making this issue to discuss an idea. In the 'old days' we had a compiled C++ version of the firmware code that ran on linux using the portduino layer. We used that binary in the CI build of the python toolchain to do integration testing - because we could talk to it like a "real" device (because it was 95% the same). This was handy for integration test purposes.

I guess that somewhere over time that got deprecated/removed.

Now that there is slick codecoverage support that someone added (@ianmcorvidae ?) for python: what do you think about using that again?

The reason I ask is that in cleaning up my power measurement stuff for checkin I realized that the PowerStress stuff needs code coverage testing. I could learn how to use the python mocking framework, but I'm worried that even if I use that framework if I mock out the PowerStress class to pretend to be the real device I'm not really testing that much. Because PowerStress is mostly done on the device side.

Instead: (proposal)

• Use a precompiled linux device firmware (do ya'll already/still do a linux build of the device firmware?), fire it up.
• Run the python tests as integration tests but have them talk via TCP to the actual device firmware fired up previously.
• This would allow testing lots more of the python code - and also implicitly testing the device code as well. So it would be a true integration test.
• Testing all this code would then be seen by the code coverage tool.

Does this seem plausible? Would you like changes?

My current queue is:

• finishing the power stuff (done next few days?)
• stress testing nrf52 filesystem stuff to try and reproduce the reported rare corruption and make a robust fix (idk, a couple of weeksish?)
• do this third?

[ianmcorvidae]

I've thought similarly, so I definitely agree! There's still a SimRadio system built off portduino, used among other things by the simulator (Meshtasticator), as well as the actual linux-native meshtasticd that can talk to a radio using SPI. So I think it's quite within reach. I'm not who added the code coverage testing -- I think it's nice for knowing what's being tested to some extent, but I'm not sold on the broad usefulness of striving for 100% coverage. I've worked a little bit with hypothesis, a property-based testing system (if you've ever touched Haskell -- QuickCheck is property-based testing), and I'd love if a lot of our tests could move to something like that which works more off stating assumptions and checking that they hold, with something like what you're talking about doing integration-type testing (which is what a lot of our current stuff is doing, albeit by way of a lot of mocks that aren't always that current and a lot that aren't even functional -- so something running actual firmware code would definitely be better).

As far as implicitly testing the firmware code, I think that's also good and we might even be able to build this such that it can optionally use real devices (somewhere far away from defaults/the public mesh), which could uncover device-specific bugs, potentially. I don't imagine we could systematize that without a lot of resources (especially given how many devices we support these days), but it'd be a cool option to have.

Basically: yes, I like this!

[geeksville]

ooh Meshtasticator looks great (I didn't even know this hotness existed - awesome work @GUVWAF !) and is probably an indirect descendent of that old linux build I was mentioning. awesome!

[geeksville]

ooh the docker 'auto build and run the native build' container stuff is slick.

[thebentern]

I'm game! I think some of the nominal testing that was originally there shook loose from the 1.3/2.0 days when we moved all of the config protos around.

Long term, I'd actually like to have real IoT hardware in play for e2e testing. I think ESP32 based targets could work the easiest, since we have esptool access already in python. I wonder if a USB hub with programmatically switchable ports exists, because inevitably when something goes wrong, power cycling could save us some flakey test heartache.

[GUVWAF]

While I'm not proficient with Python testing, I'm happy to help with some Python glue code to create a basic test scenario of 3 Portduino nodes that talk to each other with a hop in between. I believe with that you can cover a lot of test cases for mesh communication (broadcasts, DMs with ACK, traceroute, etc.) already.

[ianmcorvidae]

Hi @GUVWAF -- I was thinking of digging into this a little bit soon possibly. I can probably adapt things to work with pytest fixtures if you're able to give me an idea of the setup code that needs to be run (and the constraints on it -- I guess primarily that these will need to run on Linux and not other platforms). So when you've got a chance, let me know -- we can chat in Discord or such too if that would be helpful. I'm thinking that with something simple in place I can get us some simple behavior tests (send DM A -> C, confirm C gets it and sends back an ack to A; send traceroute and check how the route is shown, etc.) to get this process started. Thanks for the offer of help; sorry to get back to it so late!

[GUVWAF]

Hey @ianmcorvidae, in short you'll indeed have to run it on a Linux machine, or in a Docker container. You'll have to run multiple instances of the program, all listening to a different TCP port as specified with -p. Then you open a connection to each instance and subscribe to simulator packets. Then, depending on who transmitted a packet (you can map it to a node based on the TCP port you received it on), you can decide who will receive this and forward the packet to those nodes. And then you can just use the connections as normal to send DMs and check who received it, etc.

Hope this helps to start, feel free to ask more questions here or on Discord.