#

subword-tokenization

Here are 5 public repositories matching this topic...

DolbyUUU / byte_pair_encoding_BPE_subword_tokenization_implementation_python

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

python nlp natural-language-processing tokenizer data-preprocessing data-cleaning bpe byte-pair-encoding subword-tokenization

Updated Jan 30, 2023
Python

SD7Campeon / Comment-Toxicity-Detection-and-Classification

LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.

nlp sklearn transformer discourse-analysis multi-label-classification affective-computing keras-tensorflow text-vectorization bilstm nlp-pipeline deep-sequential-model toxicity-analysis toxicity-prediction toxicity-detection toxicity-classification llm subword-tokenization real-time-inference contextual-nlp

Updated May 15, 2025
Jupyter Notebook

moralesangel / BPE-tokenizer

A clean, educational implementation of the Byte Pair Encoding algorithm used in modern language models like GPT.

nlp machine-learning natural-language-processing deep-learning tokenizer transformers text-processing gpt tokenization bpe byte-pair-encoding large-language-models llm generative-ai subword-tokenization

Updated Nov 11, 2025
Jupyter Notebook

shivendrra / shredword-trainer

Fast & Efficient BPE & Unigram tokenizer trainer library

natural-language-processing tokenizer tokenization subword-tokenization

Updated Nov 1, 2025
C++

FatimaALzahrani / Byte-Pair-Encoding-Demo

A minimal Python implementation of Byte Pair Encoding (BPE) with step-by-step visualization of merge operations and vocabulary updates.

nlp natural-language-processing tokenization bpe byte-pair-encoding subword-tokenization

Updated Oct 21, 2025
HTML

Improve this page

Add a description, image, and links to the subword-tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the subword-tokenization topic, visit your repo's landing page and select "manage topics."