Be consistent with span logging, set error=true and use otlog.Error

[cstyan]

It's very confusing to look at a trace UI for two traces that seemed to have failed for the same reason, yet one has error ! icons and the other doesn't. This seems to be a side effect of not setting error=true on all error cases, and one case where we had error="a string".

There's also a change required in upstream weaveworks/common to add another error=true

Signed-off-by: Callum Styan callumstyan@gmail.com

cc @gouthamve @gotjosh @joe-elliott

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

Isn’t this better solved by consistently using spanlogger that we use elsewhere?

[cstyan]

Good point, looks like spanlogger Error via opentracings ext.Error.Set(s.Span, true) should solve most of this for us.

[pracucci]

@cstyan Thanks for fixing this. I left few comments.

Could you fix the linter, please? For the error Error return value of log.Error is not checked (errcheck) you could just add it to the ignore list defined in .errcheck-exclude (we don't want to check errors from logger)

[pracucci]

New span here. Should replace the context, no?

[cstyan]

I'm basing this off how we used the ot library sp, _ := opentracing.StartSpanFromContext(ctx, "ParseQueryRangeResponse"), where we ignored the returned context. I don't know the historical reason behind doing it that way, so if you think it should change here with the introduction of spanlogger that's fine by me.

[pracucci]

I think the reason is that ctx is not used below within this function. I believe replacing it is more future proof to avoid mistakes (we should do it once we'll start using ctx below and it's easy to forget to do it).

[pracucci]

Isn't the previous message "wrong number of samples" more correct? It's also what we log.

[pracucci]

Same comment here

[cstyan]

Thanks for the review @pracucci, I've replied to the comments that are still open, let me know what you think.

[cstyan]

@pracucci can you let me know your preferences re: my replies to your comments?

[pracucci]

Thanks @cstyan for patiently addressing my feedback 🙏 Your comment about logging is right. I left few minor comments and then we should be good to go.

[pracucci]

I believe we shouldn't repeat this nolint comment wherever we use log.Error() but add it to .errcheck-exclude. What do you think?

[cstyan]

I couldn't get that to work properly, for various variations of (import path.Struct).Function. If you have a suggestion for what to use as the entry in that file let me know.

[cstyan]

Please see the latest commit as well, if we want to have the ctx assignment in the bigtable index client for example, we need nolint comments again.

[pstibrany]

I think (*github.com/cortexproject/cortex/pkg/util/spanlogger.SpanLogger).Error should work (it does with standalone errcheck utility). When I update golangci-lint in the build image to 1.30.0, I no longer see this warning, even without changing .errcheck-exclude. 😕

We could also remove error return value from this method, since no production code uses it.

[pracucci]

I think the reason is that ctx is not used below within this function. I believe replacing it is more future proof to avoid mistakes (we should do it once we'll start using ctx below and it's easy to forget to do it).

[pstibrany]

While doing this change, would it make sense to move error-marking to Log method? If passed level is error, we would mark span with error flag, and if there is any error object passed, we would wrap first one into otlog.Error.

This would also make (*SpanLogger).Error(error) method unnecessary.

(I thought it already works this way when making my earlier suggestion, but that doesn't seem to be the case)

[cstyan]

Thanks for the review again @pracucci I think everything is updated correctly now. There's just the lint exclusion left, I was not able to get an entry into the file that worked locally, which is why I went with the nolint comments.

@pstibrany are you referring to the spanloggers Log function? That's just a wrapper for go-kit/log's Log, which has no notion of log level since that's done via the go-kit/log/levels wrappers. So there isn't actually any Log(msg, level) that could take error and log to error level and add the correct error tag to the span. Think we should just stick with compatibility with the logger we use.

[pstibrany]

@pstibrany are you referring to the spanloggers Log function? That's just a wrapper for go-kit/log's Log, which has no notion of log level since that's done via the go-kit/log/levels wrappers. So there isn't actually any Log(msg, level) that could take error and log to error level and add the correct error tag to the span. Think we should just stick with compatibility with the logger we use.

Yes, I'm referring to Log function. Log level is passed as just another key-value pair to that method (key being level.Key() and value being level.ErrorValue() for errors), and Log method is already processing all key-value pairs inside, so it can as well understand this special combination. I think it's worth exploring.

[cstyan]

Yes, I'm referring to Log function. Log level is passed as just another key-value pair to that method (key being level.Key() and value being level.ErrorValue() for errors), and Log method is already processing all key-value pairs inside, so it can as well understand this special combination. I think it's worth exploring.

Oh interesting, TIL. Good find, yeah we could then check kvps for a level=error combination and set the tag on the span. The question would be whether we want to assign all kvps as fields on the span as tags or log messages, or just look for a second field err or error.

Are you okay if I just open a new issue to look into this, instead of doing it as part of this PR? Having the error tags in general is going to help with debugging the ruler already, which is what prompted me to make this change.

[pstibrany]

Oh interesting, TIL. Good find, yeah we could then check kvps for a level=error combination and set the tag on the span. The question would be whether we want to assign all kvps as fields on the span as tags or log messages, or just look for a second field err or error.

Log method already adds all kvps as fields to the span, but doesn't mark span with error flag, and doesn't use correct key for error value.

Are you okay if I just open a new issue to look into this, instead of doing it as part of this PR? Having the error tags in general is going to help with debugging the ruler already, which is what prompted me to make this change.

👍

[pstibrany]

I think (*github.com/cortexproject/cortex/pkg/util/spanlogger.SpanLogger).Error should work (it does with standalone errcheck utility). When I update golangci-lint in the build image to 1.30.0, I no longer see this warning, even without changing .errcheck-exclude. 😕

We could also remove error return value from this method, since no production code uses it.

[pracucci]

@cstyan I've pushed a commit to try to globally exclude SpanLogger.Error() from errcheck (it works on my local machine, let's see what CI says...).

[pracucci]

LGTM! @pstibrany do you have any other comment?