10

I am trying to create chunks (max) 350 characters long with 100 chunk overlap.

I understand that chunk_size is an upper limit, so I may get chunks shorter than that. But why am I not getting any chunk_overlap?

Is it because the overlap also has to split on one of the separator chars? So it's 100 chars chunk_overlap if there is a separator within 100 chars of the split that it can split on?

from langchain.text_splitter import RecursiveCharacterTextSplitter some_text = """When writing documents, writers will use document structure to group content. \ This can convey to the reader, which idea's are related. For example, closely related ideas \ are in sentances. Similar ideas are in paragraphs. Paragraphs form a document. \n\n \ Paragraphs are often delimited with a carriage return or two carriage returns. \ Carriage returns are the "backslash n" you see embedded in this string. \ Sentences have a period at the end, but also, have a space.\ and words are separated by space.""" r_splitter = RecursiveCharacterTextSplitter( chunk_size=350, chunk_overlap=100, separators=["\n\n", "\n", "(?<=\. )", " ", ""] ) x = r_splitter.split_text(some_text) print(x) for thing in x: print(len(thing))

Output

["When writing documents, writers will use document structure to group content. This can convey to the reader, which idea's are related. For example, closely related ideas are in sentances. Similar ideas are in paragraphs. Paragraphs form a document.", 'Paragraphs are often delimited with a carriage return or two carriage returns. Carriage returns are the "backslash n" you see embedded in this string. Sentences have a period at the end, but also, have a space.and words are separated by space.'] 248 243
Improve this question
2
  • 1
    Did you figure this out?I have the same question Commented Aug 4, 2023 at 16:00
  • Unfortunately not Commented Jan 30, 2024 at 13:34

2 Answers 2

Reset to default
11

I found that RecursiveCharacterTextSplitter will not overlap chunks that are split by a separator, like how you have it: separators=["\n\n", "\n", "(?<=\. )", " ", ""]

What's happening is that each of your two paragraphs is being made into its own whole chunk due to the \n\n separator. Thus these chunks are considered separate and will not generate overlap. If you had a paragraph that was greater than your 350 chunk size (or if your chunk size was smaller), the paragraph would get split into multiple chunks, and those chunks would have overlap.

I assume the package's logic is that, since you are purposefully semantically separating those paragraphs, you wouldn't want them to have their messages overlapped. If it is something you want, I'd recommend removing the relevant separators.

Note: My answer breaks down a little when you consider that " " is a separator as well. You'd think that would make each word its own chunk. I don't understand that part yet.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Yeah it does break down a bit when you think about " "...
This was very informative. I was wondering why text that had been extracted from a PDF using PyPDF was handled so poorly. Turned out that I needed to join the sentences that made up each paragraph to remove hard linebreaks, and then the splitting worked correctly with overlap.
0

The RecursiveCharacterTextSplitter is not very good at getting nice overlaps. However it will always try to overlap the chunks, if possible. The overlap is decided by the size of the splits of the separator. So if the first separator does a good split (all chunks less than chunk_size), the others will not be used to get a finer overlap split.

For example:
You have chunk_size=500 and overlap=50
The first separator splits the document into 7 chunks with the following lengths:
[100, 300, 100, 100, 100]
The chunks will then be merged together until the next split in line will exceed the limit.
So chunks 0, 1, and 2 will be merged together to form a final document chunk. Since the last chunk in the merge (chunk 2), has size 100 which is bigger than the allowed overlap it will not be included in the next merge of chunks 3 and 4 thus giving no overlap.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.