Bye Bye Elites 👋 ChatGPT Levels the Playing Field in Search Performance and UX vs Google Search

by

👨‍💻 TLDR: Following OpenAI’s ChatGPT integration into Bing in February 2023, Bing’s traffic increased by 15.8%, while Google’s declined by 1%, a consequential shift considering each 1% change in search market share equates to $2 billion. A recent study compared user behaviors using Google’s keyword-based method and ChatGPT’s conversational approach, involving 95 participants. Despite ChatGPT’s slower response and misinformation issues, users spent less time on tasks without hindering overall performance. Although it struggled with fact-checking tasks, ChatGPT was great with straightforward questions and leveled user performance across various education levels unlike Google. Users rated ChatGPT’s information quality higher, expressed similar trust levels for both tools, and found ChatGPT more useful and satisfying.

In an intriguing new study, scientists set out to determine how users behave differently when seeking information through traditional search engines versus artificial intelligence-powered chatbots like ChatGPT. The digital experiment showed that the chatbot users managed to complete tasks in less time without hurting the task performance.

Interestingly, ChatGPT seemingly bridged the performance divide between various educational levels. While it effectively answered simple questions and provided basic solutions, it ran into challenges in tasks requiring fact-checking.

Here’s the graphic from the paper describing the performance on two information-gathering tasks comparing five different educational levels from high school to doctorate (highlights by me):

Notes: I added two red and two blue lines, red for the performance increase of the ChatGPT group with increasing education level and blue for the performance increase of the Google Search group. A vertical bar (e.g., for ChatGPT) means that a high-school student can produce the same output quality as a doctorate degree on the specific task at hand.

Despite these hiccups, the users deemed ChatGPT’s responses superior in quality to Google Search, and placed equal trust in both platforms. They also reported better overall experience with the chatbot, finding it enjoyable and satisfying. However, researchers warn of potential risks such as overreliance and misinformation propagation through the chatbot.

My Personal View

These insights could be game-changing for search engine development, suggesting the possibility of weaving in chatbot technology for a richer user experience.

Google search has had time to optimize for speed and user experience whereas ChatGPT still has many low-hanging fruits to be deployed from a UX standpoint:

  • Improve speed by 10x so ChatGPT will feel much more intuitive and responsive.
  • Integrate with many apps (e.g., using Plugins) and browsers so ChatGPT will be 10x more useful.
  • Improve the chatbot performance (e.g., integrating real-time data) so ChatGPT output quality will increase by 10x.

Taken together, these and other improvements can easily result in multiple orders of magnitude of performance increase so ChatGPT will not only be faster and produce better quality results but it will also feel much more natural!

✅ Recommended: Free ChatGPT Prompting Cheat Sheet (PDF)