Skip to content

[relay] Fix stack overflow in device_planner observed on windows due … - #10124

Closed
mei-ye wants to merge 4 commits into
apache:mainfrom
mei-ye:main
Closed

mei-ye wants to merge 4 commits into
apache:mainfrom
mei-ye:main

Conversation

@mei-ye

@mei-ye mei-ye commented Feb 1, 2022

Copy link
Copy Markdown
Contributor

…to recursive function calls.

@masahi

Stack overflow are observed when large models are compiled on windows. This is due to recursive function calls in many places. This patch fixes one such place. A test is added to generate a big enough graph to expose this problem.

@tmoreau89

Copy link
Copy Markdown
Contributor

CC @rkimball - given that this affects windows support

@tmoreau89

Copy link
Copy Markdown
Contributor

Also CC @tkonolige

@mbrookhart

Copy link
Copy Markdown
Contributor

Is it possible to use the MixedModeVisitor instead of manually creating an imperative stack here? cc @mbs-octoml

@mei-ye

mei-ye commented Feb 1, 2022

Copy link
Copy Markdown
Contributor Author

Is it possible to use the MixedModeVisitor instead of manually creating an imperative stack here? cc @mbs-octoml

Thanks for the pointer to MixedModeVisitor. It appears to me that MixedModeVisitor does something like:
foreach node in PDFS order {
do(..)
}

The device_planner does something like:
foreach node visited {
do_1()
visit_child_nodes()
do_2()
}
unless we can merge do_1 into do_2, the semantics is not equivalent to MixeModeVisitor.

@masahi masahi self-assigned this Feb 1, 2022
@jroesch

jroesch commented Feb 1, 2022

Copy link
Copy Markdown
Member

cc @mbs-octoml

// We don't care if the call is in pre- or post-lowered form.
auto vanilla_call = GetAnyCall(call_node);

// Find the higher-order domain for the callee. See DomainForCallee for the special rules

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please preserve the comments? Thanks!

Comment thread src/relay/transforms/device_planner.cc Outdated
<< domains_->ToString(func_domain) << std::endl
<< "and implied call virtual devices:" << std::endl
<< domains_->ToString(implied_domain);
std::stack<Expr> stack;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote this as a "naive" ExprVisitor and not as a MixedModeVisitor since I wrongly assumed it would only see post-ANF graphs. Hence the iterative rather than recursive handling of LetNodes but arbitrary recursion everywhere else. So sorry this is causing you trouble!

Since it is just setting up domain constraints the order of visiting shouldn't matter much (at least for valid Relay graphs) and there's no tricky state management inside the visitor. So I think this could be converted to a MixedModeVisitor pretty easily. Would you be comfortable trying to do that? Happy to help review that, offer help, etc.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have time to rewrite this pass as MixedModeVisitor? I am time-stretched,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened a bug: #10135
Could you help to fix it? I can help to test the fix on windows.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can also do it when I have time.

@mei-ye
mei-ye requested a review from Huyuwei as a code owner February 8, 2022 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants