[ML] Forecast results show non intuitive trends

[pheyos]

Found in version

• 6.3.0-bc11

Steps to reproduce

1. Create a multi metric ML job (e.g. for the farequote dataset with 15m buckets and mean(responsetime) partition_field_name=airline)
2. Create a 28d forecast on the results

Actual result
The forecast shows a trend to infinity or zero in an exponential manner, which is sometimes unexpected when looking at the actual data the forecast is based on. And it is a significant change compared to the results of the same forecast in version 6.2.5:

Additional information
In other datasets with more base data for the forecast (e.g. cloudwatch) the same trend can be seen:

[tveasey]

This is definitely undesirable. The usage is a bit unusual: i.e. forecasting multiples of the full data range used to generate the model. Also, these signals are not really interesting from a forecasting perspective: basically constant + (white) noise. However, I'd expect much closer to fixed level extrapolation in this case. This needs understanding and likely indicates a bug. (In 6.2 we wouldn't in fact model a trend at all for such signals, which is causing the difference.)

[tsviel]

Elasticsearch version (Elastic cloud deployment v6.6.0):

Description of the problem including expected versus actual behavior:
Irrational forecasts were made to for given time series data.
Given the data it's easy for the naked eye to see that the forecasts are irrational given the history in the following examples:

Series 1: https://drive.google.com/file/d/1kOwXnQaYp1AcuC9SlZoe_LyQsVryLQd8/view?usp=sharing
Series 2: https://drive.google.com/file/d/1bWqHNXjF_UdLWcWeOYmpCyLkBYgABoK9/view?usp=sharing
Series 3: https://drive.google.com/file/d/1tq6prsKzqhsKQNZai7kIkG4mIqSagjhw/view?usp=sharing

Steps to reproduce:

1. Create index

PUT example_data/
{
"mappings": {
"doc": {
"properties": {
"ts":{
"type": "date",
"format": "epoch_second"
},
"val": {
"type": "long"
},
"series_id": {
"type": "keyword"
}}}}}

2. Index the attached example_data.json
   https://drive.google.com/file/d/18cnS-hr26kIC44KEsYT5mLwVoyI3wAWF/view?usp=sharing
3. Create ml job with the following ml job json configuration.
   https://drive.google.com/open?id=1GiegY2TSgF527zYlUfO_uIM6blidy2Me
4. Create forecast for 8 weeks.

[tveasey]

Thanks for the additional data sets. I've been busy on other work, but this is starting to rise in importance again.

All these problem cases have a feature in common: forecasts are made with after supplying relatively little data and often the forecast interval is (considerably) longer than the data observed.

One change which is going to help matters is that we'll simply be more cautious in making forecasts with relatively little history (#214) and stop early. However, as before I want to understand why the trend model is so much in error in compared to what one would visually expect and also critically IMO why the predicted uncertainties aren't larger in these cases.

[tsviel]

Thank you for you comment.

I strongly disagree with the assumption that in most cases the forecast interval is longer than the data observed since i'v seen in it many time series with 3x the forecast interval.

[tveasey]

The original set of problem cases were fixed by #463. Also 2 of the series in this comment are now ok. I'd like to close this issue in preference of a more targeted one covering the remaining problem, which I now understand better. @pheyos can you take a look at 7.1 when the fix is available and confirm.

[pheyos]

@tveasey I've checked on 7.1 and the examples given in the original description are looking good again in my test environment.
👍 for closing this issue.

[tveasey]

Thanks @pheyos I'll close.