The robot sample program has a bug

[yolo-hyl]

I run the

python agents/robot_traj/app.py

but the Gradio's website does not have any output.
The console terminal output is

expected bytes, NoneType found

I try to print the model's output, find this:

Mark 6 at [251,92], Mark 8 at [250,22] should be moved, and their future positions are:

"Mark 6": "[[252,90],[251,87],[247,86],[241,87],[232,89],[221,95],[211,101],[202,107]]"

"Mark 8": "[[251,20],[250,17],[247,16],[240,18],[230,23],[218,32],[206,43],[196,54]]"

The analysis is correct.
Debug found that the error appeared in robot_traj/utils/visualier.py

Traceback (most recent call last): File "/home/user/nlp/Magma/agents/robot_traj/test.py", line 52, in <module> vis.visualize(video, overlay_traces, overlay_visibility) File "/home/user/nlp/Magma/agents/robot_traj/utils/visualizer.py", line 127, in visualize self.save_video(res_video, filename=filename, writer=writer, step=step) File "/home/user/nlp/Magma/agents/robot_traj/utils/visualizer.py", line 151, in save_video video_writer.append_data(frame) File "/home/user/.conda/envs/magma/lib/python3.10/site-packages/imageio/v2.py", line 226, in append_data return self.instance.write(im, **self.write_args) File "/home/user/.conda/envs/magma/lib/python3.10/site-packages/imageio/plugins/pyav.py", line 674, in write self.init_video_stream(codec, fps=fps, pixel_format=out_pixel_format) File "/home/user/.conda/envs/magma/lib/python3.10/site-packages/imageio/plugins/pyav.py", line 886, in init_video_stream stream = self._container.add_stream(codec, fps) File "av/container/output.pyx", line 61, in av.container.output.OutputContainer.add_stream File "av/codec/codec.pyx", line 86, in av.codec.codec.Codec.__cinit__ TypeError: expected bytes, NoneType found
How to solve this problem? Thank you

[jwyang]

thanks for sharing this, I also met this before, let me check.

[jwyang]

hy (@yolo-hyl) , I addressed this issue by reinstalling cotracker from the source code, can you try it on your side?

Also, make sure you have the following version of imageio:

Name: imageio
Version: 2.35.1
Summary: Library for reading and writing a wide range of image, video, scientific, and volumetric data formats.
Home-page: https://github.com/imageio/imageio
Author: imageio contributors
Author-email: almar.klein@gmail.com
License: BSD-2-Clause

[rr3087]

hi Jianwei Yang (@jwyang)
I actually built co-tracker from source and then updated the imageio library to 2.35.1
But I am still getting that error and so it does not produce an output video

Thx :)

[jwyang]

rr3087 Interesting, I encountered this issue a while ago, and then fixed it somehow. Can you try to install pyav with the following version and see?

Name: pyav
Version: 14.2.1
Summary: Pythonic bindings for FFmpeg's libraries.
Home-page: https://pyav.basswood-io.com
Author: 
Author-email: WyattBlue <wyattblue@auto-editor.com>
License: BSD-3-Clause

[rr3087]

already have that version installed

[yolo-hyl]

Jianwei Yang (@jwyang)
me too. Like this rr3087. my conda env pip :

Package                      Version     Editable project location
---------------------------- ----------- ---------------------------
absl-py                      2.1.0
accelerate                   0.34.2
aiofiles                     23.2.1
aiohappyeyeballs             2.4.6
aiohttp                      3.11.13
aiosignal                    1.3.2
albucore                     0.0.23
albumentations               2.0.5
altair                       5.5.0
annotated-types              0.7.0
anthropic                    0.49.0
anyio                        3.7.1
array_record                 0.6.0
astor                        0.8.1
astunparse                   1.6.3
async-timeout                5.0.1
attrs                        25.1.0
azure-core                   1.32.0
azure-identity               1.20.0
beautifulsoup4               4.13.3
bitsandbytes                 0.44.1
blinker                      1.9.0
boto3                        1.37.10
botocore                     1.37.10
cachetools                   5.5.2
certifi                      2025.1.31
cffi                         1.17.1
cfgv                         3.4.0
charset-normalizer           3.4.1
click                        8.1.8
colorama                     0.4.6
comtypes                     1.4.10
contourpy                    1.3.1
cotracker                    3.0         /home/weihua/nlp/co-tracker
cryptography                 44.0.2
cycler                       0.12.1
Cython                       3.0.12
dashscope                    1.22.2
datasets                     3.3.2
decorator                    5.2.1
deepspeed                    0.16.4
defusedxml                   0.7.1
dill                         0.3.8
distlib                      0.3.9
distro                       1.9.0
dlimp                        0.0.1
dm-tree                      0.1.9
docker-pycreds               0.4.0
draccus                      0.10.0
easyocr                      1.7.2
einops                       0.6.1
einops-exts                  0.0.4
etils                        1.12.0
evaluate                     0.4.3
exceptiongroup               1.2.2
fastapi                      0.115.11
ffmpy                        0.5.0
filelock                     3.17.0
fire                         0.7.0
flash_attn                   2.7.4.post1
flatbuffers                  25.2.10
fonttools                    4.56.0
frozenlist                   1.5.0
fsspec                       2024.12.0
ftfy                         6.3.1
gast                         0.6.0
gitdb                        4.0.12
GitPython                    3.1.44
google-auth                  2.38.0
google-auth-oauthlib         1.2.1
google-pasta                 0.2.0
gradio                       4.44.1
gradio_client                1.3.0
groq                         0.18.0
grpcio                       1.70.0
h11                          0.14.0
h5py                         3.13.0
hjson                        3.1.0
httpcore                     1.0.7
httpx                        0.28.1
huggingface-hub              0.29.1
identify                     2.6.9
idna                         3.10
imageio                      2.35.1
importlib_resources          6.5.2
iniconfig                    2.0.0
Jinja2                       3.1.5
jiter                        0.9.0
jmespath                     1.0.1
joblib                       1.4.2
jsonschema                   4.22.0
jsonschema-specifications    2024.10.1
keras                        2.15.0
kiwisolver                   1.4.8
latex2mathml                 3.77.0
lazy_loader                  0.4
libclang                     18.1.1
lightning-utilities          0.12.0
llvmlite                     0.44.0
lmdb                         1.6.2
loguru                       0.7.3
lxml                         5.3.1
magma                        0.0.1       /home/weihua/nlp/Magma
Markdown                     3.7
markdown-it-py               3.0.0
markdown2                    2.5.3
MarkupSafe                   2.1.5
matplotlib                   3.10.1
mdurl                        0.1.2
mergedeep                    1.3.4
ml-dtypes                    0.2.0
modelscope                   1.23.2
MouseInfo                    0.1.3
mpmath                       1.3.0
msal                         1.31.1
msal-extensions              1.2.0
msgpack                      1.1.0
multidict                    6.1.0
multiprocess                 0.70.16
mypy-extensions              1.0.0
narwhals                     1.30.0
networkx                     3.4.2
ninja                        1.11.1.3
nodeenv                      1.9.1
numba                        0.61.0
numpy                        1.26.4
nvidia-cublas-cu12           12.4.5.8
nvidia-cuda-cupti-cu12       12.1.105
nvidia-cuda-nvrtc-cu12       12.1.105
nvidia-cuda-runtime-cu12     12.1.105
nvidia-cudnn-cu12            8.9.2.26
nvidia-cufft-cu12            11.0.2.54
nvidia-curand-cu12           10.3.2.106
nvidia-cusolver-cu12         11.4.5.107
nvidia-cusparse-cu12         12.1.0.106
nvidia-ml-py                 12.570.86
nvidia-nccl-cu12             2.20.5
nvidia-nvjitlink-cu12        12.8.61
nvidia-nvtx-cu12             12.1.105
oauthlib                     3.2.2
open_clip_torch              2.31.0
openai                       1.3.5
opencv-contrib-python        4.11.0.86
opencv-python                4.11.0.86
opencv-python-headless       4.11.0.86
OpenEXR                      3.3.2
opt-einsum                   3.3.0
orjson                       3.10.15
packaging                    24.2
paddleocr                    2.10.0
paddlepaddle                 2.6.2
pandas                       2.2.3
peft                         0.4.0
pillow                       10.4.0
pip                          25.0.1
platformdirs                 4.3.6
pluggy                       1.5.0
portalocker                  2.10.1
pre-commit                   3.8.0
promise                      2.3
propcache                    0.3.0
protobuf                     3.20.3
psutil                       5.9.8
py-cpuinfo                   9.0.0
pyarrow                      19.0.1
pyasn1                       0.6.1
pyasn1_modules               0.4.1
PyAutoGUI                    0.9.54
pyav                         14.2.1
pyclipper                    1.3.0.post6
pycparser                    2.22
pydantic                     2.10.6
pydantic_core                2.27.2
pydeck                       0.9.1
pydub                        0.25.1
PyGetWindow                  0.0.9
Pygments                     2.19.1
PyJWT                        2.10.1
PyMsgBox                     1.0.9
pyparsing                    3.2.1
pyperclip                    1.9.0
PyRect                       0.2.0
PyScreeze                    1.0.1
pytest                       8.3.3
pytest-asyncio               0.23.6
python-bidi                  0.6.6
python-dateutil              2.9.0.post0
python-docx                  1.1.2
python-multipart             0.0.20
python3-xlib                 0.15
pytorch-lightning            2.5.0.post0
pytweening                   1.2.0
pytz                         2025.1
PyYAML                       6.0.2
pyyaml-include               1.4.1
RapidFuzz                    3.12.2
referencing                  0.36.2
regex                        2024.11.6
requests                     2.32.3
requests-oauthlib            2.0.0
rich                         13.9.4
rpds-py                      0.23.1
rsa                          4.9
ruff                         0.6.7
s3transfer                   0.11.4
sacrebleu                    2.5.1
safetensors                  0.5.3
scikit-image                 0.25.2
scikit-learn                 1.5.0
scipy                        1.15.2
screeninfo                   0.8.1
seaborn                      0.13.2
semantic-version             2.10.0
sentencepiece                0.1.99
sentry-sdk                   2.22.0
setproctitle                 1.3.5
setuptools                   75.8.0
shapely                      2.0.7
shellingham                  1.5.4
shortuuid                    1.0.13
simsimd                      6.2.1
six                          1.17.0
smmap                        5.0.2
sniffio                      1.3.1
soupsieve                    2.6
spaces                       0.32.0
sqlitedict                   2.1.0
starlette                    0.46.0
streamlit                    1.43.1
stringzilla                  3.12.3
supervision                  0.18.0
svgwrite                     1.4.3
sympy                        1.13.3
tabulate                     0.9.0
tenacity                     9.0.0
tensorboard                  2.15.2
tensorboard-data-server      0.7.2
tensorflow                   2.15.0
tensorflow-addons            0.23.0
tensorflow-datasets          4.9.3
tensorflow-estimator         2.15.0
tensorflow-graphics          2021.12.3
tensorflow-io-gcs-filesystem 0.37.1
tensorflow-metadata          1.16.1
termcolor                    2.5.0
threadpoolctl                3.5.0
tifffile                     2025.2.18
timm                         0.9.12
tokenizers                   0.19.1
toml                         0.10.2
tomli                        2.2.1
tomlkit                      0.12.0
torch                        2.3.1
torchmetrics                 1.6.2
torchvision                  0.18.1
tornado                      6.4.2
tqdm                         4.67.1
transformers                 4.44.1
trimesh                      4.6.4
triton                       2.3.1
typeguard                    2.13.3
typer                        0.15.2
typing_extensions            4.12.2
typing-inspect               0.9.0
tzdata                       2025.1
uiautomation                 2.0.20
ultralytics                  8.3.70
ultralytics-thop             2.0.14
urllib3                      2.3.0
uvicorn                      0.34.0
virtualenv                   20.29.3
wandb                        0.19.7
watchdog                     6.0.0
wavedrom                     2.0.3.post3
wcwidth                      0.2.13
websocket-client             1.8.0
websockets                   12.0
Werkzeug                     3.1.3
wheel                        0.45.1
wrapt                        1.14.1
xxhash                       3.5.0
yarl                         1.18.3
zipp                         3.21.0

i ensure my cotracker reinstall cotracker from the source code, this:

git clone https://github.com/facebookresearch/co-tracker
cd co-tracker/
pip install -e .

But still got the same error.
It may be that the array is out of bounds?
Do you have any other solution?
Thanks:)

[jwyang]

hmm, that's weird, can you try run this?

pip install imageio[ffmpeg]

I do see that I installed imageio using the above command. Also, according to ChatGPT, it seems that the root cause is from ffmpeg, and the official cotracker suggests the above installation for offline mode.

[yolo-hyl]

amazing!!!
This indeed solved my problem.

[yolo-hyl]

Jianwei Yang (@jwyang) Can it predict 3D coordinates of trajectory points, sach as (x,y,z,pitch,roll,yaw)? Is this limited to locations on the image?
The distance from the picture to the 3D world is a bit large.
Thanks:)

[jwyang]

hy (@yolo-hyl) , it is great that the bug is fixed!

This demo is for showing how Magma generates the visual trace planning for robotcis manipulation tasks. All these traces are 2D.

If you want to check the 7DoF output, please take a look at this part to set up the SimplerEnv:

https://github.com/microsoft/Magma?tab=readme-ov-file#evaluation-with-simplerenv

[jwyang]

I think this issue has been fixed, and am going to clouse this issue. thanks to everyone for raising this issue!