Fix info parameter not appearing in autograding failure messages#18
Merged
Conversation
Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com> Agent-Logs-Url: https://github.com/DataScience4Psych/actions/sessions/97f1339e-2aa6-48b1-bfe6-d0f12d6192a6
Copilot
AI
changed the title
[WIP] Add info message to unit test feedback
Fix Mar 23, 2026
info parameter not appearing in autograding failure messages
smasongarrison
marked this pull request as ready for review
March 23, 2026 19:33
smasongarrison
added a commit
that referenced
this pull request
Mar 23, 2026
…20) * Revert "Fix `info` parameter not appearing in autograding failure messages (#18)" This reverts commit 8d7a9c2. * Initial plan * Fix TAP message parsing: preserve indentation for multi-line failure messages - Store tap_lines without trimws() so indentation is retained - Update grep/sub patterns to match indented YAML message keys - Strip YAML block/folded scalar indicators (|-, |, >-, >) when they are the entire value after message: - Replace fixed '^ ' continuation check with dynamic indentation-depth comparison so continuation lines (deeper indent than message: key) are correctly collected while sibling keys like severity: are excluded Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com> Agent-Logs-Url: https://github.com/DataScience4Psych/actions/sessions/71658e60-8cda-4f29-bf56-aa97015ea3ec * Fix empty continuation lines and unify block/inline message handling Skip empty continuation lines (matching main branch behavior) and use a unified content-first-then-append pattern for both block scalars and inline messages. This makes the implementation produce identical output to main for all |- block scalar cases, while also correctly handling >- folded scalars (which main did not recognize). Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com> Agent-Logs-Url: https://github.com/DataScience4Psych/actions/sessions/c6fdd31e-b2c5-4ce2-acf4-815d4505d692 --------- Co-authored-by: Mason Garrison <garrissm@wfu.edu> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
expect_equal(and similar) failures include aninfoparameter, the "Failure message" column in grading results was blank. The TAP reporter emits multi-line messages as a YAML block scalar (|-), but the parser was callingtrimws()on every collected line, destroying indentation — leavingraw = "|-"which renders as blank in a GitHub markdown table cell.Changes
msg_lines(removedtrimws()from collection loop)grep/subpatterns to match 2-space-indented YAML keys (^[[:space:]]*message:)|-or|, read content lines until hitting a sibling key (^ [^[:space:]]), concatenating non-empty content with spaces^ [^[:space:]])Before / After
`answer_in_rmd` is not TRUE Make sure to include the number of rows in the .rmd fileOriginal prompt
This section details on the original issue you should resolve
<issue_title>adding "info" to the unit test feedback</issue_title>
<issue_description>most of the tests have additional feedback under info. it doesn't show up in the result.
test_that("Ex 1: Rmd Exercise 1 reports correct number of rows", {
skip_if(length(.rmd_content) == 0)
data("datasaurus_dozen", package = "datasauRus", envir = environment())
solution_nrow <- as.character(nrow(get("datasaurus_dozen", envir = environment())))
potential_answers <- c(solution_nrow, "nrow\(datasaurus_dozen\)", "names\(datasaurus_dozen\)")
pattern <- paste0("(", paste(potential_answers, collapse = "|"), ")")
answer_in_rmd <- stringr::str_detect(.rmd_content, pattern) |> any()
expect_equal(answer_in_rmd, TRUE,
info = "Make sure to include the number of rows in the .rmd file"
)
})
the output looks like this, but the | Failure message is blank instead of using the info
LAB11 autograding results
| Expectation | Failure message -- | -- | -- ☑ | ok 1 Rmd file contains headers | ☑ | ok 2 Rmd file contains custom header elements | ☑ | ok 3 Rmd contains a minimum number of R code chunks | ☑ | ok 4 R code chunks contain actual code (not all empty) | ✗ | not ok 1 Ex 1: Titanic data object exists | ❔ | ok 2 # SKIP Reason: No Titanic data object found | ❔ | ok 3 # SKIP Reason: No Titanic data object found | ❔ | ok 4 # SKIP Reason: No Titanic data object found | ❔ | ok 5 # SKIP Reason: No Titanic data object found | ❔ | ok 6 # SKIP Reason: No Titanic data object found | ✗ | not ok 7 Ex 1: titanic3 data object exists | ✗ | not ok 8 Ex 1: m_apparent model object exists | ❔ | ok 9 # SKIP Reason: m_apparent not found | ❔ | ok 10 # SKIP Reason: m_apparent not found | ❔ | ok 11 # SKIP Reason: m_apparent not found | ❔ | ok 12 # SKIP Reason: p_apparent not found | ❔ | ok 13 # SKIP Reason: m_apparent not found | ✗ | not ok 14 Ex 10: cv_confusion data frame exists | ❔ | ok 15 # SKIP Reason: cv_confusion not found | ❔ | ok 16 # SKIP Reason: cv_confusion not found | ❔ | ok 17 # SKIP Reason: cv_confusion not found | ❔ | ok 18 # SKIP Reason: cv_confusion not found | ❔ | ok 19 # SKIP Reason: cv_confusion not found | ❔ | ok 20 # SKIP Reason: cv_confusion not found | ✗ | not ok 21 Ex 10: Rmd contains confusion matrix code (TP, FP, TN, FN) | ✗ | not ok 22 Ex 10: Rmd discusses sensitivity vs specificity trade-off | ✗ | not ok 23 Ex 2: logistic regression mo...⌨️ Start Copilot coding agent tasks without leaving your editor — available in VS Code, Visual Studio, JetBrains IDEs and Eclipse.