The code is to demonstrate usage of streaming with GPT-4 API, ChatGPT API and InstructGPT (GPT-3.5.) models & Streamlit-app.
The approach uses only openai and time libraries and re-prints the streams using print(end='', flush=True):
!pip install --upgrade openai import openai import time openai.api_key = user_secrets.get_secret("OPENAI_API_KEY") startime = time.time()Disclaimer: The downside of streaming in production usage is the control of appropiate usage policy: https://beta.openai.com/docs/usage-guidelines, which should be reviewed in advance for each application, so I suggest to take a look this policy prior deciding to use streaming.
Run the file streams.ipnyb first part.
### STREAM GPT-4 API RESPONSES delay_time = 0.01 # faster max_response_length = 8000 answer = '' # ASK QUESTION prompt = input("Ask a question: ") start_time = time.time() response = openai.ChatCompletion.create( # GPT-4 API REQQUEST model='gpt-4', messages=[ {'role': 'user', 'content': f'{prompt}'} ], max_tokens=max_response_length, temperature=0, stream=True, # this time, we set stream=True ) for event in response: # STREAM THE ANSWER print(answer, end='', flush=True) # Print the response # RETRIEVE THE TEXT FROM THE RESPONSE event_time = time.time() - start_time # CALCULATE TIME DELAY BY THE EVENT event_text = event['choices'][0]['delta'] # EVENT DELTA RESPONSE answer = event_text.get('content', '') # RETRIEVE CONTENT time.sleep(delay_time)After inserting the user input and pressing enter, you should see the output printed:
Run the file streams.ipnyb second part. Add user input and you should see similar to below:
### STREAM CHATGPT API RESPONSES delay_time = 0.01 # faster max_response_length = 200 answer = '' # ASK QUESTION prompt = input("Ask a question: ") start_time = time.time() response = openai.ChatCompletion.create( # CHATPG GPT API REQQUEST model='gpt-3.5-turbo', messages=[ {'role': 'user', 'content': f'{prompt}'} ], max_tokens=max_response_length, temperature=0, stream=True, # this time, we set stream=True ) for event in response: # STREAM THE ANSWER print(answer, end='', flush=True) # Print the response # RETRIEVE THE TEXT FROM THE RESPONSE event_time = time.time() - start_time # CALCULATE TIME DELAY BY THE EVENT event_text = event['choices'][0]['delta'] # EVENT DELTA RESPONSE answer = event_text.get('content', '') # RETRIEVE CONTENT time.sleep(delay_time)
Run the file streams.pnyb third part. Add user input and you should see similar to below:
collected_events = [] completion_text = [] speed = 0.05 #smaller is faster max_response_length = 200 start_time = time.time() prompt = input("Ask a question: ") # Generate Answer response = openai.Completion.create( model='text-davinci-003', prompt=prompt, max_tokens=max_response_length, temperature=0, stream=True, # this time, we set stream=True ) # Stream Answer for event in response: event_time = time.time() - start_time # calculate the time delay of the event collected_events.append(event) # save the event response event_text = event['choices'][0]['text'] # extract the text completion_text += event_text # append the text time.sleep(speed) print(f"{event_text}", end="", flush=True)
I add a working "app_streamlit.py"-file, which you can fork to your repository with the "requirements.txt" and deploy it in Streamlit.
In the advanced settings, add the OPENAI_API_KEY-variable using format:
OPENAI_API_KEY = "INSERT HERE YOUR KEY"Feel free to fork and further improve the code as per the license. For example you can further improve the ChatML to ensure the flow follows desired "system" rules. I left these empty now to make this basic script very generic. I recommend to check my articles specific to ChatGPT API about streaming responses in Medium related to Streaming, ChatML: guiding prompts with system, assistant and user roles and ChatGPT API introduction tutorial.