cChardet

cChardet is high speed universal character encoding detector. - binding to uchardet.

This fork (`faust-cchardet`) vs `PyYoshi/cChardet`

This is the faust-streaming maintained fork of upstream PyYoshi/cChardet. It exists mainly to keep the project building on current Python and Windows toolchains and to publish up-to-date prebuilt wheels across platforms. The public Python API is unchanged from upstream.

The distribution is renamed on PyPI; the import name is not:

pip install faust-cchardet

import cchardet   # same import name as upstream

Key differences

Prebuilt wheels built with cibuildwheel, so most users install without a C++ compiler:
- Linux: x86_64, i686, aarch64 (CPython and PyPy)
- macOS: x86_64, arm64
- Windows: x86_64 / AMD64
Windows / MSVC support — pinned to an MSVC-compatible uchardet (bdb8a0…) so the C++ extension compiles under Microsoft Visual C++. Upstream's newer uchardet header does not build under MSVC.
Modern build — uses the meson-python build backend (distutils is removed in Python 3.12+); Meson builds the Cython output as C++ and links the matching C++ runtime automatically.
Versioning — the version lives in src/cchardet/version.py and is exposed as cchardet.__version__ (upstream uses setuptools_scm).
Python support — requires Python >= 3.10 (3.6–3.9 are dropped).

Detection differences (a consequence of the pinned uchardet)

A UTF-8 byte-order mark is reported as UTF-8-SIG (upstream reports UTF-8).
On 32-bit (i686) builds a few near-equivalent labels differ — e.g. Thai TIS-620 is detected as ISO-8859-11.

Building against a system `uchardet`

By default cChardet builds a bundled copy of uchardet (vendored as a git submodule), which is what the published wheels ship. Distributions and source builds can instead link the system uchardet through the system-uchardet Meson feature option:

`-Dsystem-uchardet=`	Behaviour
`disabled` (default)	Always build the bundled copy (used by the wheels).
`enabled`	Require the system library; the build fails if it is missing or too old. Intended for distro packaging — read the detection-quality caveat below first.
`auto`	Use the system library if it is new enough, otherwise fall back to the bundled copy.

Pass the option through your build front-end. With pip / build the meson-python backend reads it from setup-args:

pip install . --config-settings=setup-args=-Dsystem-uchardet=enabled

Requirement: the system uchardet must be recent enough to expose uchardet_get_n_candidates. That symbol currently ships only in the git version (https://gitlab.freedesktop.org/uchardet/uchardet.git) — no tagged release provides it yet — so enabled/auto probe for it and reject an older library.

Warning

A system build currently detects non-UTF-8 input less accurately than the bundled build. The bundled copy compiles a cChardet-specific replacement for uchardet's multi-byte group prober, which both restores detection throughput and rejects non-UTF-8 byte sequences that upstream reports as UTF-8. A system build links upstream's prober, so neither applies.

Measured over 1650 non-UTF-8 documents spanning 7 encodings (benchmarks/make_nonutf8_corpus.py, median of 3 runs):

Build	Throughput	Non-UTF-8 reported as UTF-8
bundled (wheel default)	3.10 MB/s	0.0%
system `uchardet`	1.44 MB/s	16.2%

Reporting non-UTF-8 bytes as UTF-8 is the defect that caused v3.0.0 to be yanked from PyPI, so this is not a cosmetic difference. It is an upstream issue rather than a packaging mistake: nsUTF8Prober does not reject invalid sequences on its own, and its confidence never falls low enough for the candidate to be discarded. Patches have been sent upstream; once they land, the build can require a uchardet version that includes them and this caveat goes away.

If you are packaging cChardet for a distribution that forbids bundled libraries, consider carrying the overlay (src/cchardet/uchardet-overlay/nsMBCSGroupProber.cpp) as a patch against your system uchardet until then. The corpus generator and benchmark above are in-tree, so you can verify the result yourself.

Supported Languages/Encodings

International (Unicode)
- UTF-8
- UTF-16BE / UTF-16LE
- UTF-32BE / UTF-32LE / X-ISO-10646-UCS-4-34121 / X-ISO-10646-UCS-4-21431
Arabic
- ISO-8859-6
- WINDOWS-1256
Bulgarian
- ISO-8859-5
- WINDOWS-1251
Chinese
- ISO-2022-CN
- BIG5
- EUC-TW
- GB18030
- HZ-GB-2312
Croatian:
- ISO-8859-2
- ISO-8859-13
- ISO-8859-16
- Windows-1250
- IBM852
- MAC-CENTRALEUROPE
Czech
- Windows-1250
- ISO-8859-2
- IBM852
- MAC-CENTRALEUROPE
Danish
- ISO-8859-1
- ISO-8859-15
- WINDOWS-1252
English
- ASCII
Esperanto
- ISO-8859-3
Estonian
- ISO-8859-4
- ISO-8859-13
- ISO-8859-13
- Windows-1252
- Windows-1257
Finnish
- ISO-8859-1
- ISO-8859-4
- ISO-8859-9
- ISO-8859-13
- ISO-8859-15
- WINDOWS-1252
French
- ISO-8859-1
- ISO-8859-15
- WINDOWS-1252
German
- ISO-8859-1
- WINDOWS-1252
Greek
- ISO-8859-7
- WINDOWS-1253
Hebrew
- ISO-8859-8
- WINDOWS-1255
Hungarian:
- ISO-8859-2
- WINDOWS-1250
Irish Gaelic
- ISO-8859-1
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
Italian
- ISO-8859-1
- ISO-8859-3
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
Japanese
- ISO-2022-JP
- SHIFT_JIS
- EUC-JP
Korean
- ISO-2022-KR
- EUC-KR / UHC
Lithuanian
- ISO-8859-4
- ISO-8859-10
- ISO-8859-13
Latvian
- ISO-8859-4
- ISO-8859-10
- ISO-8859-13
Maltese
- ISO-8859-3
Polish:
- ISO-8859-2
- ISO-8859-13
- ISO-8859-16
- Windows-1250
- IBM852
- MAC-CENTRALEUROPE
Portuguese
- ISO-8859-1
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
Romanian:
- ISO-8859-2
- ISO-8859-16
- Windows-1250
- IBM852
Russian
- ISO-8859-5
- KOI8-R
- WINDOWS-1251
- MAC-CYRILLIC
- IBM866
- IBM855
Slovak
- Windows-1250
- ISO-8859-2
- IBM852
- MAC-CENTRALEUROPE
Slovene
- ISO-8859-2
- ISO-8859-16
- Windows-1250
- IBM852
- MAC-CENTRALEUROPE

Example

One-shot detection

import cchardet as chardet

with open(r"src/tests/samples/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt", "rb") as f:
    msg = f.read()
    result = chardet.detect(msg)
    print(result)

Streaming detection

import cchardet

detector = cchardet.UniversalDetector()
with open("bigfile.txt", "rb") as f:
    for line in f:
        detector.feed(line)
        if detector.done:
            break
detector.close()
print(detector.result)

Thread safety

cchardet.detect() is safe to call concurrently from any number of threads. Each call creates its own uchardet detector, uses it, and frees it before returning, so no state is shared between calls.

A UniversalDetector instance is not safe to share across threads. An instance holds the state of a single stream -- the native uchardet handle, the completion flags, and the leading bytes kept for BOM detection -- and feed(), close(), reset() and result all mutate it. Give each thread its own detector, or guard a shared one with your own lock:

import cchardet
from concurrent.futures import ThreadPoolExecutor

def sniff(path):
    # one detector per call, so nothing crosses a thread boundary
    with cchardet.UniversalDetector() as detector:
        with open(path, "rb") as f:
            for line in f:
                detector.feed(line)
                if detector.done:
                    break
        return detector.result

with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(sniff, paths))

On the free-threaded builds of CPython (3.13t / 3.14t) the _cchardet extension declares itself free-threading compatible, so importing cchardet does not re-enable the GIL for the whole process. The rules above do not change on those builds. Sharing one detector across threads still gives meaningless results, but it cannot corrupt the interpreter: the instance methods take a per-instance critical section, which costs nothing on ordinary GIL builds.

Note that this buys correctness, not extra parallelism -- detection itself still runs while holding the GIL on non-free-threaded builds.

Command line

A cchardetect console script is installed with the package:

$ cchardetect src/tests/samples/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt
src/tests/samples/wikipediaJa_One_Thousand_and_One_Nights_SJIS.txt: SHIFT_JIS with confidence 0.99

$ cat somefile.txt | cchardetect        # also reads from stdin

Benchmark

$ pip install -e .
$ python src/tests/bench.py

Results

CPU: AMD Ryzen 9 7950X3D

RAM: DDR5-5600MT/s 96GB

Platform: Ubuntu 24.04 amd64

Python 3.12.3

	Request (call/s)
chardet v5.2.0	1.1
cchardet v2.2.0a1	2263.6

LICENSE

See COPYING file.

Contact

Issues

Support Platforms

Prebuilt wheels are published for:

Windows x86_64 (AMD64)
Linux x86_64, i686, aarch64
macOS x86_64, arm64

Name		Name	Last commit message	Last commit date
Latest commit History 355 Commits
.github		.github
benchmarks		benchmarks
dockerfiles		dockerfiles
src		src
.gitignore		.gitignore
.gitmodules		.gitmodules
CHANGES.rst		CHANGES.rst
COPYING		COPYING
Makefile		Makefile
Pipfile		Pipfile
Pipfile.lock		Pipfile.lock
README.md		README.md
TODO.md		TODO.md
meson.build		meson.build
meson.options		meson.options
pyproject.toml		pyproject.toml
requirements-dev.lock		requirements-dev.lock
requirements-dev.txt		requirements-dev.txt
requirements.lock		requirements.lock
tox.ini		tox.ini

Navigation Menu

Uh oh!

Repository files navigation

cChardet

This fork (`faust-cchardet`) vs `PyYoshi/cChardet`

Building against a system `uchardet`

Supported Languages/Encodings

Example

One-shot detection

Streaming detection

Thread safety

Command line

Benchmark

Results

Python 3.12.3

LICENSE

Contact

Support Platforms

About

Releases

Packages

Used by

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

cChardet

This fork (faust-cchardet) vs PyYoshi/cChardet

Building against a system uchardet

Supported Languages/Encodings

Example

One-shot detection

Streaming detection

Thread safety

Command line

Benchmark

Results

Python 3.12.3

LICENSE

Contact

Support Platforms

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

This fork (`faust-cchardet`) vs `PyYoshi/cChardet`

Building against a system `uchardet`