Argparse wrapping is bugged when colors are involved

[alexprengere]

Bug report

Bug description:

While working on coloring interpolated values in #141940, I realized that in some cases the wrapping is broken.

For example in this case, just a color change around the commas makes the wrapping different, because textwrap.wrap is sensitive to ANSI escape codes (those change the string length).

There are 2 ways to fix this:

• have a more clever wrapping that is not sensitive to ANSI escape codes
• make sure the wrapping always happens before ANSI escape codes are inserted

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-142035: Textwrap without ansi escapes #142040
• gh-142035: Make TextWrapper ANSI-aware #152702

[picnixz]

cc @hugovk @savannahostrowski

[alexprengere]

Looking a bit more at the code, I believe it would make more sense to add a parameter to textwrap to ignore ANSI escape codes when wrapping/filling, then leverage it in argparse.

With the recent addition of colors in the stdlib, I think it makes sense to have that option in textwrap directly. I would argue it should default to True, but I will leave that to the experts 😉.

I opened #142040 as a proof-of-concept, if this is viable for the code owners, I can add the tests, docs, etc.

EDIT: this is also an issue with multiple-code-points glyphs, textwrap is not handling them cleanly, so it can cause other wrapping issues:

>>> textwrap.wrap("Cafe\u0301 is good", width=4)
['Cafe', '́ is', 'good']
>>> textwrap.wrap("Cafe\u0301 is good", width=5)
['Café', 'is', 'good']

AFAICT there is no clean way in stdlib to get the actual rendering width of such strings, like what is done by third party libs like wcwidth.

[hugovk]

Please can you share some example code that causes this?

[alexprengere]

Running on the current CPython main:

import argparse

parser = argparse.ArgumentParser(
    description="A simple argument parser example.",
    formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
parser.add_argument("--verbose", action="store_true", help="A l o n g d e s c r i p t i o n   f o r   t h e   v e r b o s e   f l a g   t o   d e m o n s t r a t e   t h e   f o r m a t t e r   c l a s s   u s e d   i n   t h i s   a r g u m e n t   p a r s e r .")
parser.add_argument("--input", type=str, default="input.txt", help="Input file path")
args = parser.parse_args()

Then:

$ COLUMNS=70 python test.py --help
usage: test.py [-h] [--verbose] [--input INPUT]

A simple argument parser example.

options:
  -h, --help     show this help message and exit
  --verbose      A l o n g d e s c r i p t i o n f o r t h e v e r b
                 o s e f l a g t o d e m o n s t r a t e t h e f o r
                 m a t t e r c l a s s u s e d i n t h i s a r g u m
                 e n t p a r s e r . (default:
                 False)
  --input INPUT  Input file path (default:
                 input.txt)

You can see the False for --verbose is not wrapping correctly.
If you modify the script to manually write the default value, so it does not have the colors introduced in #141680

import argparse

parser = argparse.ArgumentParser(
    description="A simple argument parser example.",
)
parser.add_argument("--verbose", action="store_true", help="A l o n g d e s c r i p t i o n   f o r   t h e   v e r b o s e   f l a g   t o   d e m o n s t r a t e   t h e   f o r m a t t e r   c l a s s   u s e d   i n   t h i s   a r g u m e n t   p a r s e r . (default: False)")
parser.add_argument("--input", type=str, default="input.txt", help="Input file path")
args = parser.parse_args()

Here is the text is wrapping correctly.

usage: test.py [-h] [--verbose] [--input INPUT]

A simple argument parser example.

options:
  -h, --help     show this help message and exit
  --verbose      A l o n g d e s c r i p t i o n f o r t h e v e r b
                 o s e f l a g t o d e m o n s t r a t e t h e f o r
                 m a t t e r c l a s s u s e d i n t h i s a r g u m
                 e n t p a r s e r . (default: False)
  --input INPUT  Input file path

#142040 is one way to fix it.

[picnixz]

Maybe we could assume that (default: ...) is actually added to the description text before we format the entire thing? It looks like we are not wrapping the same text.

[yihong0618]

Looking a bit more at the code, I believe it would make more sense to add a parameter to textwrap to ignore ANSI escape codes when wrapping/filling, then leverage it in argparse.

With the recent addition of colors in the stdlib, I think it makes sense to have that option in textwrap directly. I would argue it should default to True, but I will leave that to the experts 😉.

I opened #142040 as a proof-of-concept, if this is viable for the code owners, I can add the tests, docs, etc.

EDIT: this is also an issue with multiple-code-points glyphs, textwrap is not handling them cleanly, so it can cause other wrapping issues:

textwrap.wrap("Cafe\u0301 is good", width=4)
['Cafe', '́ is', 'good']
textwrap.wrap("Cafe\u0301 is good", width=5)
['Café', 'is', 'good']
AFAICT there is no clean way in stdlib to get the actual rendering width of such strings, like what is done by third party libs like wcwidth.

wcwidth in terminal is hard.....
pdm also found similar issue when color the arg when update to 3.14
info: pdm-project/pdm#3682

[alexprengere]

For the record, as I explained in the linked PR, I believe the best solution to fix this would be to add a way to customize the length function of textwrap, like what was proposed in #28136.
argparse could then use this to avoid counting the escape codes when wrapping.

[picnixz]

I would be happy to have a custom length because I needed this in Sphinx. If I have more time this week-end I'll try to ressurect that other PR and check whether we can do something with this. This is a bit unfortunate that this will only be present in 3.15+ though, considering it's kinda annoying in Python 3.14. It's very user-facing in general now that all CLIs are colored (I don't know if they are colored by default but I think so) so we should also find a way to make it work in 3.14.