Python: fix compaction token count inflating non-ASCII text

[he-yufeng]

Motivation and Context

TokenBudgetComposedStrategy estimates a message's token count by feeding its JSON serialization to the tokenizer. _serialize_message() built that JSON with ensure_ascii=True, which escapes non-ASCII text into \uXXXX sequences. The tokenizer then counts the escape sequences instead of the characters the model actually tokenizes — e.g. こんにちは is counted as こんにちは. For non-Latin content this inflates the estimate (~1.6x for a mixed Japanese message with the built-in CharacterEstimatorTokenizer, and more for pure CJK), so compaction and token-budget decisions fire on numbers that diverge from real LLM usage.

Description

Serialize with ensure_ascii=False so non-ASCII text is token-counted as the characters the model sees. This matches the ensure_ascii=False already used elsewhere in the same module, and it only affects token estimation — the serialized string is never stored or transmitted. Added a regression test that asserts the serialized payload keeps the real characters (no \uXXXX) and that its token count is lower than the escaped form's.

Related Issue

Fixes #7022

Contribution Checklist

• The code builds and tests pass locally (pytest tests/core/test_compaction.py, ruff format + check clean)
• I have added a test that proves the fix is effective
• The PR is linked to an issue with a closing keyword
• The change is focused and does not touch unrelated code

[Copilot]

Pull request overview

This PR fixes token-estimation inflation for non-ASCII (e.g., CJK) messages in Python compaction by ensuring message serialization used for token counting does not escape Unicode into \uXXXX sequences, bringing estimates closer to real model tokenization behavior and addressing #7022.

Changes:

• Updated _serialize_message() to use json.dumps(..., ensure_ascii=False) so token counting operates on actual Unicode characters.
• Added a regression test that verifies serialized payload retains non-ASCII characters and yields a lower token count than an escaped representation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
python/packages/core/agent_framework/_compaction.py	Switches message serialization for token estimation to ensure_ascii=False to avoid inflated counts from escaped Unicode.
python/packages/core/tests/core/test_compaction.py	Adds a regression test covering non-ASCII serialization and confirming token count reduction vs an escaped form.