Hindi Extraction Issue Summary

[krish-tech02]

• Library / Build

  • PyMuPDF (fitz) 1.24.x
  • Windows 11, Python 3.12 (venv writtenpro)
  • Attachment: Addiction, Understanding the Disease of 01-12-2025_Hindi_Andovar - Copy.pdf

• Repro Steps

  import fitz, sys sys.stdout.reconfigure(encoding="utf-8") doc = fitz.open(r"Addiction, Understanding the Disease of 01-12-2025_Hindi_Andovar - Copy.pdf") page = doc[0] print(page.get_text()[:300])

Addiction, Understanding the Disease of 01-12-2025_Hindi_Andovar - Copy.pdf

 - **Observed Output (PyMuPDF)** 

व्यसन एक स्ववस्थ्य समस्यव है जो तब होती है जब कोई व्यक्तत ककसी पदार्थ, जैसे ड्रग्स या शराब, का सेवन बंद नहीं कर पाता...

Multiple syllables are duplicated (`व्यक्तत`, `ककसी`, `मस्स्तष्क`, etc.). - **Expected Text (from original DOCX / visual PDF copy‑paste)** 

व्यसन एक स्वास्थ्य समस्या है जो तब होती है जब कोई व्यक्ति किसी पदार्थ, जैसे ड्रग्स या शराब, का सेवन बंद नहीं कर पाता...

 - **Font Table (fitz `page.get_fonts(full=True)`)** 

(5, 'ttf', 'TrueType', 'BCDEEE+ArialMT', 'F1', 'WinAnsiEncoding', 0)
(9, 'ttf', 'Type0', 'BCDFEE+NirmalaUI', 'F2', 'Identity-H', 0)
(14,'ttf', 'Type0', 'BCDGEE+Noto Sans Devanagari,Bold', 'F3', 'Identity-H', 0)
(19,'ttf', 'TrueType', 'BCDHEE+Noto Sans Devanagari,Bold', 'F4', 'WinAnsiEncoding', 0)
(21,'ttf', 'Type0', 'BCDIEE+Noto Sans Devanagari', 'F5', 'Identity-H', 0)
(26,'ttf', 'TrueType', 'BCDJEE+Noto Sans Devanagari', 'F6', 'WinAnsiEncoding', 0)
...

 The affected text is emitted by fonts `F4` / `F6` (TrueType + `WinAnsiEncoding`) even though they are Devanagari fonts. Those spans have `span["chars"]` with `cid=None`, so PyMuPDF appears to rely on the WinAnsi map, yielding incorrect Unicode. - **Docx Source Verification** We unpacked `word/document.xml`; Hindi runs do not contain the duplicated characters, so corruption happens during PDF export or decoding. - **Problem Statement for Support** “When we extract text from the attached Hindi PDF, spans using `Noto Sans Devanagari` that are tagged as `WinAnsiEncoding` produce garbled syllables—each consonant is replaced by a different letter pair. Spans using the same font but exported as `Type0 / Identity-H` decode correctly. It seems PyMuPDF falls back to the WinAnsi code page because `cid` is `None`, but the glyphs are Devanagari. Is there an option or workaround to recover the correct Unicode in this situation (e.g., forcing glyph extraction, using ToUnicode if present, or ignoring the WinAnsi tag when the font name is non-Latin)?” - **Request** 1. Confirmation if PyMuPDF treats `TrueType + WinAnsiEncoding` fonts strictly as Latin and cannot derive proper Unicode for Devanagari. 2. Guidance on any API that could map these glyphs back to Unicode (e.g., using raw glyph IDs, switching to `textpage.extractRAWDICT`, or an undocumented flag). 3. Suggestions for detecting the issue programmatically, so we can alert authors or attempt a fallback. Any insights or recommended workarounds from the PyMuPDF team will help us decide whether we need a preprocessing step before extraction or if Word must be prevented from emitting WinAnsi subsets for Hindi. [Addiction, Understanding the Disease of 01-12-2025_Hindi_Andovar - Copy.pdf](https://github.com/user-attachments/files/23724252/Addiction.Understanding.the.Disease.of.01-12-2025_Hindi_Andovar.-.Copy.pdf) @JorjMcKie Please help 

[krish-tech02]

[krish-tech02]

Addiction, Understanding the Disease of 01-12-2025_Hindi_Andovar - Copy.pdf

[krish-tech02]

@JorjMcKie could please help here?