Fixed #1153 Track Random Seed

[seanlaw]

Fixed #1153

Also see #1112

Pull Request Checklist

Below is a simple checklist but please do not hesitate to ask for assistance!

• Read our Contributing Guide
• Referenced a Github issue (or create one if one doesn't already exist)
• Left a meaningful comment on the original Github issue to discuss the detailed approach for your contribution
• Forked, cloned, and checkedout the newest version of the code
• Created a new branch
• Made necessary code changes
• Installed black (i.e., python -m pip install black or conda install -c conda-forge black)
• Installed flake8 (i.e., python -m pip install flake8 or conda install -c conda-forge flake8)
• Installed pytest-cov (i.e., python -m pip install pytest-cov or conda install -c conda-forge pytest-cov)
• Ran black --exclude=".*\.ipynb" --extend-exclude=".venv" --diff ./ in the root stumpy directory
• Ran flake8 --extend-exclude=.venv ./ in the root stumpy directory
• Ran ./setup.sh dev && ./test.sh in the root stumpy directory

[gitnotebooks]

Review these changes at https://app.gitnotebooks.com/stumpy-dev/stumpy/pull/1154

[seanlaw]

Currently, tests/test_floss.py::test_cac is failing because the stumpy/floss.py::_iac function is modified to use our custom rng.RNG:

    with rng.fix_seed(seed):
        I = rng.RNG.randint(0, width, size=width, dtype=np.int64)
        if bidirectional is False:  # Idealized 1-dimensional matrix profile index
            I[:-1] = width
            for i in range(width - 1):
                I[i] = rng.RNG.randint(i + 1, width, dtype=np.int64)

        target_AC = _nnmark(I)

        params = np.empty((n_iter, 2), dtype=np.float64)
        for i in range(n_iter):
            hist_dist = scipy.stats.rv_histogram(
                (target_AC, np.append(np.arange(width), width))
            )
            data = hist_dist.rvs(size=n_samples)
            a, b, c, d = scipy.stats.beta.fit(data, floc=0, fscale=width)

            params[i, 0] = a
            params[i, 1] = b

However, internally, the hist_dist uses the default np.random random number generator to sample from the desired histogram distribution rather than using our custom rng.RNG. To remedy this, we can force hist_dist to use our custom rng.RNG by adding hist_dist.random_state = rng.RNG and the result looks like:

with rng.fix_seed(seed):
        I = rng.RNG.randint(0, width, size=width, dtype=np.int64)
        if bidirectional is False:  # Idealized 1-dimensional matrix profile index
            I[:-1] = width
            for i in range(width - 1):
                I[i] = rng.RNG.randint(i + 1, width, dtype=np.int64)

        target_AC = _nnmark(I)

        params = np.empty((n_iter, 2), dtype=np.float64)
        for i in range(n_iter):
            hist_dist = scipy.stats.rv_histogram(
                (target_AC, np.append(np.arange(width), width))
            )
            hist_dist.random_state = rng.RNG
            data = hist_dist.rvs(size=n_samples)
            a, b, c, d = scipy.stats.beta.fit(data, floc=0, fscale=width)

            params[i, 0] = a
            params[i, 1] = b

[seanlaw]

With our new setup for tracking the random seed, at the start of each unit test file, you'll see:

conftest.py: rng.set_seed(3032615292)

So, to reproduce all of the tests, you simply need to go into the conftest.py file and replace the random seed inside of:

def pytest_configure(config):
    """
    Called after command line options have been parsed
    and all plugins and initial conftest files been loaded.
    """
    # rng.set_seed(seed)  # Replace this with your failed unit test seed

    print(f"conftest.py: rng.set_seed({rng.SEED})")

This should make reproducing failed tests locally a lot easier! And then revert the conftest.py file when you're done (i.e., remove the seed).

Note that all of this is still using the old "legacy numpy random number generator" and so all of our tests (that require a specific seed) are unchanged.