Skip to content

[App] Restore cpp_rtvm native inference tool on Relax VM and the tvm-ffi runtime - #20484

Closed
Raman-Raje wants to merge 2 commits into
apache:mainfrom
Raman-Raje:rtvm
Closed

Raman-Raje wants to merge 2 commits into
apache:mainfrom
Raman-Raje:rtvm

Conversation

@Raman-Raje

Copy link
Copy Markdown
Contributor

Summary

This PR brings back apps/cpp_rtvm, the standalone C++ tool (rtvm) for running TVM-compiled
models directly on a target device such as Android (OpenCL), without Python. It was removed in
#17660 along with the Relay C++ components. This version is rewritten for the current stack:

  • Relax VM executables (vm_load_executable, vm_initialization, set_input,
    invoke_stateful, get_output) instead of the graph executor
  • tvm::ffi APIs (ffi::Module::LoadFromFile, ffi::Function, Tensor)
  • Per-backend runtime libraries (TVM_RUNTIME_BACKEND_LIBS), linked the same way as apps/cpp_rpc

Build with -DUSE_CPP_RTVM=ON (the option already exists in cmake/config.cmake).

Testing

  • Built rtvm and libtvm_runner for CPU, GPU, and Android targets.
  • Verified end-to-end inference on Android devices with the OpenCL and Vulkan backends.

@srkreddy1238 srkreddy1238 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Raman-Raje

Copy link
Copy Markdown
Contributor Author

@tqchen

@tqchen

tqchen commented Sep 29, 2026

Copy link
Copy Markdown
Member

given we are moving towards making the core more lightweight and now it is reasonably easy for agent to bring up the support, i feel it is nice to actually not having to maintain applications on the core repo

@Raman-Raje

Copy link
Copy Markdown
Contributor Author

Thanks @tqchen, that makes sense given the direction of keeping the core lightweight.

@tqchen tqchen closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants