DevNet to attempts for more stable startup of Hyperspace relay

[dzmitry-lahoda]

Motivation

For some reason there is some conflict amid DevNet process which break Hyperspace relay amid Picasso Centauri.

Approach is add more split in time and other way to avoid running things too fast or at same time,

Suggested Solution

0. Needs fix(ibc): clarified connection ids in devnet #4040
1. Split out Hypersace bash from Process compose so it is more reusable and can be mixed to keep sanity.
2. Start relaying Picasso Centauri only after Channels for Composable Picasso formed.
3. Add RPC check to READY probe that Picasso and Composable got at least 5 blocks each (1 minute) produced. So nothing starts on them until.
4. In case if folder with genesis and data exists any Chain (basically random restarts in case of arbitrary who knows why crashes)
   4.1 WRITE BOLD warning that that this may be bad.
   4.2. do not write Genesis again for Centauri but try run from that. I may be the case that it start scratch and tries to create some things.
   4.3 (no part of this PR). Would be ideal to refactor to start conditionally Zombienet from existing data for same case. But that is harder.
5. cosmos init fails sometimes because 5 seconds sometimes not enough to wait for nonce, so need to track manuall
   5.1 request latest, than ++ on each call

Alternatives

Not sure, just simly plug timeouts, but timeouts like that never work stably. I never seen that, saw 100 times people tried. I tried once, and found the only fix is to be correct.

Notes

SPLIT it into 2 items.

[blasrodri]

What about using hyperspace only? No need to depend on Hermes.

[dzmitry-lahoda]

@blasrodri will it fix issue without introducing sleeps/timeouts/delays?
above is just efficiemt stable way to do so. also making random crashes of chains not so bad(seems like some instability in some chains)

[dzmitry-lahoda]

if we are sure that using only hyperspace will solve issue than sure?

so for local cosmos debug i use hermes as it is fastest and simplest setup for doing ibc testing of cw contracts.

[dzmitry-lahoda]

@kkast @blasrodri we need alignment:

1. try sleeps and reogrs - @kkast started that, it may work and be fast solution
2. try more organized option similar to 1
3. also fix crashes (reuse data folders) - optional, depending on how many crashes happen on start - in 10 runs is see may be 1 on various machines
4. use hyperspace

[blasrodri]

if we are sure that using only hyperspace will solve issue than sure?

so for local cosmos debug i use hermes as it is fastest and simplest setup for doing ibc testing of cw contracts.

It's just because me remove a dependency. Only due to that.

[dzmitry-lahoda]

we aready have devnet-xc-dotsama-fresh without osmosis and hermes. can check if that works well. here dependency already removed.

hermes is maintained by these people https://github.com/informalsystems/cosmos.nix#relayers so not really a lot of maintaince go into, given it is one command to setup all.

[dzmitry-lahoda]

@kkast so now to get end to end you run with delay or manually? right?

may you remind steps.

i will see if can replicated these with minimal delay as improvement.

[dzmitry-lahoda]

SRE failed to get DevNet running on big machine, both via nix and via docker

https://composablefinance.slack.com/archives/C05KUR705DJ/p1691612355148219

Some underlying components are unstable.

I run both ok, also found that after may be 6 hours of run, because of amount of logs and things accrued or some OOM, docker crashes and restarts. OOM will mean some underlying components are unstable.

From optimization, will add flag 1. not to restart on fail, so any fail on start will be stoppage. so SRE can provide logs at that. 2. there is already flag not to start relays, so one can see how stable things if no relaying.

So again, all is ok on CI and my machine, devnet code is good, underlying components are bad. Need special handling for this quircks. But given I run well on AWS without issues - all stable enough, not sure what is SRE struggle to run devnet server...

[dzmitry-lahoda]

also i see cosmos has export state cli, and osmosis has example even forking from testnet.
similar exists in dotsama.

so fix could be, as I mentioned early, prebake state on running machines, without running final relay, and than boot devnet from that and just launch relayers.

for me it is most forward strategy tbh

[dzmitry-lahoda]

here is my prio for devnet, as developing without this is hard #4052

i may switch to picasso after swap on osmosis done if that would be prio and improve devnet. afaik devnet runs and not blocker, so sometimes need restart and some machines are not suit who know why, but codespace work. no ideal, but that what we have.

[github-actions]

stale-issue

[github-actions]

close-issue