I encountered a significant delay in the response times on OpenAI’s GPT-3.5 Turbo API as compared to the ChatGPT — and I’m not alone with this (source). The latency is particularly noticeable when using certain accounts, with the delay making the API almost unusable for real-time applications.
Solution Idea: Experiment with Difference User Accounts to Prevent Throttling
Here’s what happens if you ask for a full dictionary of a certain word using two different accounts from the same user at the same time (left fast, right slow):
This suggests that the issue might be account-specific or related to allocating server resources per account. It is likely that OpenAI experiments with various throttling ideas to improve the efficiency of resource allocation — we all have experienced times when demand for OpenAI’s models was too much to handle.
Prompt Engineering with Python and OpenAI
You can check out the whole course on OpenAI Prompt Engineering using Python on the Finxter academy. We cover topics such as:
- Semantic search
- Web scraping
- Query embeddings
- Movie recommendation
- Sentiment analysis
👨💻 Academy: Prompt Engineering with Python and OpenAI
While working as a researcher in distributed systems, Dr. Christian Mayer found his love for teaching computer science students.
To help students reach higher levels of Python success, he founded the programming education website Finxter.com that has taught exponential skills to millions of coders worldwide. He’s the author of the best-selling programming books Python One-Liners (NoStarch 2020), The Art of Clean Code (NoStarch 2022), and The Book of Dash (NoStarch 2022). Chris also coauthored the Coffee Break Python series of self-published books. He’s a computer science enthusiast, freelancer, and owner of one of the top 10 largest Python blogs worldwide.
His passions are writing, reading, and coding. But his greatest passion is to serve aspiring coders through Finxter and help them to boost their skills. You can join his free email academy here.