Skip to content

Improve tiktoken.encoding_for_model with better option#185

Merged
hauntsaninja merged 1 commit intoopenai:mainfrom
xhluca:patch-1
Feb 9, 2024
Merged

Improve tiktoken.encoding_for_model with better option#185
hauntsaninja merged 1 commit intoopenai:mainfrom
xhluca:patch-1

Conversation

@xhluca
Copy link
Contributor

@xhluca xhluca commented Sep 4, 2023

tiktoken.encoding_for_model is a convenient function for automatically selecting models. However, the following will result in an error:

tiktoken.encoding_for_model("gpt-2") tiktoken.encoding_for_model("gpt-3.5")

This is because the only available options right now are:

tiktoken.encoding_for_model("gpt2") # Notice the lack of dash tiktoken.encoding_for_model("gpt-3.5-turbo") # gpt-3.5 is a common shorthand here

This PR will allow the use of "gpt-2" and "gpt-3.5"

@hauntsaninja
Copy link
Collaborator

Thank you!

@hauntsaninja hauntsaninja merged commit 84d88dc into openai:main Feb 9, 2024
@xhluca xhluca deleted the patch-1 branch February 9, 2024 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants