Don't fold source after an f-string with an unmatched literal bracket

[gadievron]

Description

simplify.real_code() folds all source after an f-string that contains an
unmatched literal bracket (e.g. f"[{x}", f"({n} items", an ANSI escape
f"\x1b[K{x}", an escaped brace f"{{", or a format spec like f"{x:>[}").

real_code deliberately leaves f-strings unblanked so their inner-expression
offsets stay valid. The subsequent implicit-continuation pass _parens then
counts the f-string's literal brackets too, so parens goes positive and
never returns to zero, and every newline to EOF is blanked. Everything below the
f-string collapses onto one logical line, so any offset-based analysis of that
code is wrong.

real_code's only consumer is worder, so the user-visible effect lands on
worder's header predicate: for a class-attribute def whose def follows
the f-string, find_definition resolves to the wrong location (or the call site)
and Rename silently renames the usages while leaving the def untouched (or
vice-versa) — no exception, no diagnostic. Module-level and nested defs are
immune (they don't go through that predicate).

>>> from rope.base import simplify
>>> simplify.real_code('f"[{x}"\ndef g():\n    return 1\n')
'f"[{x}" def g():     return 1 '   # 3 newlines collapsed to spaces

Fix

_blank_fstring_literal_brackets() tokenizes the source (PEP 701, Python 3.12+)
and blanks every ()[]{} character that is not covered by an OP token —
i.e. the literal/format-spec brackets, and bracket characters that are literal
data inside a nested string. Real expression brackets (f"{[1,2][0]}",
f"{x:{width}}") are OP tokens and are left untouched. Blanking is
length-preserving (one char → one space), so all offsets are unchanged; it is
required so the literal bracket stops swaying _parens. On Python < 3.12 an
f-string is one opaque STRING token and can't be split, so the pass is a strict
no-op; a TokenError/SyntaxError fallback leaves the source unchanged on
incomplete/invalid mid-edit buffers.

Tests are version-gated with only_for_versions_higher("3.12") and cover each
trigger class above plus an end-to-end find_definition + Rename regression, a
set of preservation controls (real brackets untouched), a module-level immunity
control, and the invalid-input fallback.

Known limitation

The blanking pass is gated on rope's existing regex-based f-string detection
(ignored_regions), which recognises single-line and triple-quoted f-strings but
not single-quoted multiline f-strings (f"[{1 +⏎2}"); those still fold as
before. Dropping the gate (tokenizing unconditionally) would close the remaining
case — happy to do that here instead if you'd prefer.

Related

Sibling of the still-open #499 (patched_ast sets region[0]=None on an
unmatched bracket in an f-string): same trigger family, but a different code path
(patched_ast does not call real_code). This PR does not fix #499. Same
desync class as the false-triple-quote bug #248 (fixed by #309).

---

Checklist

• I have added tests that prove my fix is effective
• I have updated CHANGELOG.md

[yangfan-yf-yf]

I reproduced two remaining correctness paths on 896c632.

First, the stated single-quoted multiline limitation is valid Python 3.12 syntax and still triggers the original corruption:

f"[{1 +
2}"
def g(): pass

compile() accepts this source, but real_code() reduces its three newlines to zero. In the equivalent class-method case, find_definition() then fails inside patched AST processing. Since this is another PEP 701 spelling of the reported trigger, I think it needs to be handled in this change rather than left as a known limitation.

Second, tokenizing the whole source makes an unrelated incomplete suffix disable the fix for an earlier valid f-string:

f"[{x}"
def g(): pass
(

The trailing open parenthesis raises TokenError, _blank_fstring_literal_brackets() returns the entire source unchanged, and the three newlines again collapse to zero. Rope routinely sees incomplete buffers while they are being edited, so the fallback should preserve the brackets already classified before the error, or the scan should be scoped so later invalid syntax cannot undo an earlier correction.

There is also a measurable cost from tokenizing the full module on each uncached real_code() call. Using rope/refactor/extract.py (41,840 characters) with 20 unique inputs over five trials, current master took 0.048–0.051 seconds and this head took 0.249–0.297 seconds, roughly 5–6× slower. Please scope the token scan to the relevant f-string spans or otherwise avoid a full-module tokenization for every detected f-string.

The focused simplify and find-definition suites pass (40 passed), and the covered single-line/triple-quoted cases behave correctly. The two correctness cases above need regression coverage before this is safe to merge.