Skip to content

short test summary and -vv #12713

Description

@ewjoachim

a detailed description of the bug or problem you are having

(I've done my best to try and see if this is a duplicate and I'm sorry if it is, but it seems it's not)

Following #11777 (8.2.0) | changelog:

Text is no longer truncated in the short test summary info section when -vv is given.

The short summary can become completely impractical with -vv because each failure message could be hundreds of lines long.

It seems to be a strange choice given #11777 is about the text being truncated in:

E       assert 50 == 49
E        +  where 50 = len({0: 0, 1: 1, 2: 2, 3: 3, ...})

and I don't see the original poster complaining about the short summary, but both were changed?

Was it the intended change ? Is there something we can do to keep the original -vv behaviour for the test output, and keep the "short test summary" short ? It's actually quite useful in CI that the last thing printed is the list of tests that failed and not why each of them failed: it's a summary and the full output is available above anyway.

I think the decision a lot of versions ago to add an excerpt of the failure reason to the short summary was a good compromise: it used otherwise unused space to add more useful info. But if that excerpt start taking more screen real estate space (potentially 100x more) than the summary itself, I'm not sure I understand why we would want that?

Maybe we could introduce a different "letter" (or a different option or something) to -r to get the list of tests without any kind of explanation of why they failed? Maybe the fix for #11777 should have affected the "failure explanation" and not the short summary?

output of pip list from the virtual environment you are using

$ pip list
pip list
Package   Version
--------- -------
iniconfig 2.0.0
packaging 24.1
pip       24.0
pluggy    1.5.0
pytest    8.3.2

pytest and operating system versions

pytest==8.3.2
Ubuntu 22.04.4 LTS

minimal example if possible

def test_foo():
    assert "a\n" * 50 == ""
$ pytest -vv
============================================================== test session starts ===============================================================
platform linux -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0 -- /home/joachim/other-src/pytest-verbose/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /home/joachim/other-src/pytest-verbose
collected 1 item                                                                                                                                 

test_verbose.py::test_foo FAILED                                                                                                           [100%]

==================================================================== FAILURES ====================================================================
____________________________________________________________________ test_foo ____________________________________________________________________

    def test_foo():
>       assert "a\n" * 50 == ""
E       AssertionError: assert 'a\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\n' == ''
E         
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a
E         + a

test_verbose.py:2: AssertionError
============================================================ short test summary info =============================================================
FAILED test_verbose.py::test_foo - AssertionError: assert 'a\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\na\n' == ''
  
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
  + a
=============================================================== 1 failed in 0.01s ================================================================

Activity

  1. added
    type: enhancementnew feature or API change, should be merged into features branch
    topic: reportingrelated to terminal output and user-facing messages and errors
    on Oct 30, 2024
  2. kenny-y-dev commented on Dec 3, 2024

    @kenny-y-dev
    Contributor

    I took a look at this and it seems that the fix implemented for #11777 may have misunderstood the ask in the original issue; the issue referenced -vv still truncating sequences when an int comparison assert based on len() failed, though the PR only adjusted the behavior of the short test summary.

    It appears that the issue demonstrated in #11777 is still present -- I do agree with @ewjoachim that the short test summary should still be brief at the end of the output for quick visual parsing, even if there is extended verbose output above. I'll take a deeper dive to see exactly why the sequence truncation is still occurring.

  3. added a commit that references this issue on Jan 26, 2025
    6c9939b
  4. ewjoachim commented on Jan 26, 2025

    @ewjoachim
    Author

    Oh, I had missed your PR. Just saw it now that it's merged. Thank you very much for taking the time :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic: reportingrelated to terminal output and user-facing messages and errorstype: enhancementnew feature or API change, should be merged into features branch

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions