mem: load only 15 common langdetect profiles to reduce memory

[KRRT7]

Monkey-patch langdetect's init_factory() to load only 15 high-coverage language profiles instead of all 55. The full set creates large n-gram probability arrays (word_lang_prob_map) that persist for the lifetime of the process. The subset covers >95% of web content by language share.

What happens with unsupported languages?

langdetect always returns a result — it scores text against loaded profiles using n-gram frequency comparison and picks the best match. For a document in an excluded language (e.g., Thai, Polish), langdetect returns the closest loaded profile instead of the correct one. This is the same behavior as before for the downstream code path: detect_languages() maps the langdetect result to an ISO 639-3 code for element metadata. The language metadata is informational — it does not gate OCR model selection or partitioning logic.

In practice this means:

• Included language (e.g., English, French, Japanese): correct detection, same as before
• Excluded language (e.g., Thai): langdetect returns the closest loaded profile (e.g., id for Thai text) instead of th. No crash, no empty result — just a less precise language tag in metadata.

The 15 languages were chosen to cover the vast majority of real-world document processing: en, es, ar, fr, de, it, pt, ru, ja, ko, zh-cn, zh-tw, hi, bn, id. If a deployment needs accurate metadata for additional languages, the set can be extended by adding to LANGDETECT_LANGUAGES in lang.py.

Benchmark

Measured with memray (memray run + memray stats --json), 3 rounds × 5 texts (EN/DE/FR/JP + short ASCII) through the real detect_languages() code path, Python 3.12.

detect_languages() — langdetect profile subsetting benchmark unstructured.partition.common.lang | 3 rounds x 5 texts | Python 3.12.12 Configuration Peak MB Saved % ------------------------------------------------------------ All 55 profiles 76.1MB 0.0MB 0.0% 15 common profiles 31.7MB 44.4MB 58.3%