Skip to content

Fix info parameter not appearing in autograding failure messages#18

Merged
smasongarrison merged 2 commits into
mainfrom
copilot/add-info-to-unit-test-feedback
Mar 23, 2026
Merged

Fix info parameter not appearing in autograding failure messages#18
smasongarrison merged 2 commits into
mainfrom
copilot/add-info-to-unit-test-feedback

Conversation

Copilot AI commented Mar 23, 2026

Copy link
Copy Markdown
Contributor

When expect_equal (and similar) failures include an info parameter, the "Failure message" column in grading results was blank. The TAP reporter emits multi-line messages as a YAML block scalar (|-), but the parser was calling trimws() on every collected line, destroying indentation — leaving raw = "|-" which renders as blank in a GitHub markdown table cell.

Changes

  • Preserve line indentation when collecting msg_lines (removed trimws() from collection loop)
  • Updated grep/sub patterns to match 2-space-indented YAML keys (^[[:space:]]*message:)
  • Handle YAML block scalars: when message value is |- or |, read content lines until hitting a sibling key (^ [^[:space:]]), concatenating non-empty content with spaces
  • Unified termination condition for both block-scalar and inline branches: stop at sibling YAML keys (^ [^[:space:]])

Before / After

# Test with info parameter:
expect_equal(answer_in_rmd, TRUE,
  info = "Make sure to include the number of rows in the .rmd file"
)
Expectation Failure message
Ex 1: Rmd Exercise 1 reports correct number of rows (blank)`answer_in_rmd` is not TRUE Make sure to include the number of rows in the .rmd file
Original prompt

This section details on the original issue you should resolve

<issue_title>adding "info" to the unit test feedback</issue_title>
<issue_description>most of the tests have additional feedback under info. it doesn't show up in the result.

test_that("Ex 1: Rmd Exercise 1 reports correct number of rows", {
skip_if(length(.rmd_content) == 0)
data("datasaurus_dozen", package = "datasauRus", envir = environment())
solution_nrow <- as.character(nrow(get("datasaurus_dozen", envir = environment())))

potential_answers <- c(solution_nrow, "nrow\(datasaurus_dozen\)", "names\(datasaurus_dozen\)")

pattern <- paste0("(", paste(potential_answers, collapse = "|"), ")")

answer_in_rmd <- stringr::str_detect(.rmd_content, pattern) |> any()

expect_equal(answer_in_rmd, TRUE,
info = "Make sure to include the number of rows in the .rmd file"
)
})

the output looks like this, but the | Failure message is blank instead of using the info

LAB11 autograding results

7 of 47 expectations passing — 15/100

  | Expectation | Failure message -- | -- | -- ☑ | ok 1 Rmd file contains headers |   ☑ | ok 2 Rmd file contains custom header elements |   ☑ | ok 3 Rmd contains a minimum number of R code chunks |   ☑ | ok 4 R code chunks contain actual code (not all empty) |   ✗ | not ok 1 Ex 1: Titanic data object exists |   ❔ | ok 2 # SKIP Reason: No Titanic data object found |   ❔ | ok 3 # SKIP Reason: No Titanic data object found |   ❔ | ok 4 # SKIP Reason: No Titanic data object found |   ❔ | ok 5 # SKIP Reason: No Titanic data object found |   ❔ | ok 6 # SKIP Reason: No Titanic data object found |   ✗ | not ok 7 Ex 1: titanic3 data object exists |   ✗ | not ok 8 Ex 1: m_apparent model object exists |   ❔ | ok 9 # SKIP Reason: m_apparent not found |   ❔ | ok 10 # SKIP Reason: m_apparent not found |   ❔ | ok 11 # SKIP Reason: m_apparent not found |   ❔ | ok 12 # SKIP Reason: p_apparent not found |   ❔ | ok 13 # SKIP Reason: m_apparent not found |   ✗ | not ok 14 Ex 10: cv_confusion data frame exists |   ❔ | ok 15 # SKIP Reason: cv_confusion not found |   ❔ | ok 16 # SKIP Reason: cv_confusion not found |   ❔ | ok 17 # SKIP Reason: cv_confusion not found |   ❔ | ok 18 # SKIP Reason: cv_confusion not found |   ❔ | ok 19 # SKIP Reason: cv_confusion not found |   ❔ | ok 20 # SKIP Reason: cv_confusion not found |   ✗ | not ok 21 Ex 10: Rmd contains confusion matrix code (TP, FP, TN, FN) |   ✗ | not ok 22 Ex 10: Rmd discusses sensitivity vs specificity trade-off |   ✗ | not ok 23 Ex 2: logistic regression mo...

⌨️ Start Copilot coding agent tasks without leaving your editor — available in VS Code, Visual Studio, JetBrains IDEs and Eclipse.

Copilot AI changed the title [WIP] Add info message to unit test feedback Fix info parameter not appearing in autograding failure messages Mar 23, 2026
Copilot AI requested a review from smasongarrison March 23, 2026 14:10
@smasongarrison
smasongarrison marked this pull request as ready for review March 23, 2026 19:33
@smasongarrison
smasongarrison merged commit 8d7a9c2 into main Mar 23, 2026
smasongarrison added a commit that referenced this pull request Mar 23, 2026
smasongarrison added a commit that referenced this pull request Mar 23, 2026
…20)

* Revert "Fix `info` parameter not appearing in autograding failure messages (#18)"

This reverts commit 8d7a9c2.

* Initial plan

* Fix TAP message parsing: preserve indentation for multi-line failure messages

- Store tap_lines without trimws() so indentation is retained
- Update grep/sub patterns to match indented YAML message keys
- Strip YAML block/folded scalar indicators (|-, |, >-, >) when they
  are the entire value after message:
- Replace fixed '^  ' continuation check with dynamic indentation-depth
  comparison so continuation lines (deeper indent than message: key)
  are correctly collected while sibling keys like severity: are excluded

Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com>
Agent-Logs-Url: https://github.com/DataScience4Psych/actions/sessions/71658e60-8cda-4f29-bf56-aa97015ea3ec

* Fix empty continuation lines and unify block/inline message handling

Skip empty continuation lines (matching main branch behavior) and use
a unified content-first-then-append pattern for both block scalars
and inline messages. This makes the implementation produce identical
output to main for all |- block scalar cases, while also correctly
handling >- folded scalars (which main did not recognize).

Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com>
Agent-Logs-Url: https://github.com/DataScience4Psych/actions/sessions/c6fdd31e-b2c5-4ce2-acf4-815d4505d692

---------

Co-authored-by: Mason Garrison <garrissm@wfu.edu>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: smasongarrison <6001608+smasongarrison@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

adding "info" to the unit test feedback

2 participants