[Inference] Improve RENDERING

[charlieforward9]

Detection overlays slow down as assessments go on, and the timer has a noticible re-render delay as if the memory is getting gummed up. It is difficult to guarantee dataset generation reliability with a gummy UI. Debug this and improve performance.

[charlieforward9]

TODO: Architect Mediapipe wrapper to work well with FFI API

[charlieforward9]

Looking into GetX as a simple state manager with a performance focus.

The most efficient roadmap would be to start by changing only the inference bloc to a GetX controller and testing improvements. If this works well, propagate this to all the other blocs iteratively.

[charlieforward9]

Experiment with package: bloc_concurrent

Read

[charlieforward9]

Timer now performs properly. Overlays have also improved. However, there remains potential for further performance improvements.

[charlieforward9]

@DrBubbles42

The performance of the application is my main issue right now. Struggling to work my way around. I have worked on widget tree optimizations with only slight gains. So, I am looking for feedback on my next best move...

Implementation

Video, Stopwatch, and Inference streams running in parallel. Pose Detection and its overlay painter access data from the video stream, skipping frames if it is actively inferring. Saving PNG images & detection to the DB ~2x/second asynchronously. The inference, PNG conversion & DB storing are all done on independent threads, so in theory, it is not affecting the UI thread.

Issue

"30 seconds" elapsed in the timer stream is equal to 37 seconds in real life. AKA, something is slowing the event cycle, UI-painter down. Overlays are probably closer to 10-20FPS.

Intent

Realtime pose overlays (30-60FPS)
AND
Save the image/classification/detection data at ~2/FPS.

Inventory of Options

See next comment

1. Buffer, Fastest to test, low probability of fix
Put the data into a buffer during the video stream and save it to DB after all the UI overlaying is done. Isolating the PNG conversion process and DB process from the performance problem.
2. Postprocess locally, Unsure if possible in Dart, probably is, but more work/delays for me
Do not store anything DURING the video stream, just overlay the pose inference for the WOW factor. AFTER the video has ended, send the video file to a local client function to re-run the pose detection on the video file and store it in the database from these detections.
3. Postprocess externally, Most scalable, probably slowest loading, longest dev turnaround
Similar to Option 2, but after the video has ended send the video file to a serverless function (Lambda (can be written in Python)) OR instance (if the media pipe library and video are too heavy for a lambda) to re-run the pose detection, returning the final, complete output object back to client.

MediaPipe offers a Python library that can do the pose detection.

To be 100% honest, I did not anticipate encountering so many challenges as I have up to now, and I apologize for what may seem to be a slow turnaround on my end. I am working my a** off to get this done, and know we are up to something good here because of the magnitude of battles I am facing. Good things don't come easy. God give me bigger battles. 🤲

I look forward to your feedback.

[charlieforward9]

To clarify, we can collect data now, and have all the functionality to create the dataset, but with a janky-feeling UI overlay, it is challenging to use the app and feel like it is actually working, especially with the timer inconsistency.

[charlieforward9]

I discussed this issue with some flutter experts, they said the issue has to do with the amount of times the image is being copied.

Native camera -> Raw texture copy to flutter -> Raw texture copy across isolate -> Raw texture copy for inference.

His recommendation:

If you truly want B L A Z I N G L Y F A S T performance, I'd do the camera stream, capture, and image composition natively, then just send a texture to flutter
https://api.flutter.dev/flutter/rendering/TextureLayer-class.html

This involves rebuilding (everything in the purple box) in Swift, and running it on the iOS side of the app rather than the Flutter side.

Given the magnitude of this task, I worked around it by lowering the image quality to 352x288, which is acceptable for inference as the image is processed at 255x255. However, the image preview is clearly negatively affected.

Therefore, I am making this an issue that will be addressed sometime in the future, once we have more time and resources to scale the application.