Chat#

This example will show you how to chat with GPT, the original example is on OpenAI Example, the difference is that we will teach you how to cache the response for exact and similar matches with gptcache, it will be very simple, you just need to add an extra step to initialize the cache.

Before running the example, make sure the OPENAI_API_KEY environment variable is set by executing echo $OPENAI_API_KEY. If it is not already set, it can be set by using export OPENAI_API_KEY=YOUR_API_KEY on Unix/Linux/MacOS systems or set OPENAI_API_KEY=YOUR_API_KEY on Windows systems.

Then we can learn the usage and acceleration effect of gptcache by the following code, which consists of three parts, the original openai way, the exact search and the similar search. You can also try this example on Google Colab.

OpenAI API original usage#

import time
import openai


def response_text(openai_resp):
    return openai_resp['choices'][0]['message']['content']


question = 'what‘s github'

# OpenAI API original usage
start_time = time.time()
response = openai.ChatCompletion.create(
  model='gpt-3.5-turbo',
  messages=[
    {
        'role': 'user',
        'content': question
    }
  ],
)
print(f'Question: {question}')
print("Time consuming: {:.2f}s".format(time.time() - start_time))
print(f'Answer: {response_text(response)}\n')
Question: what‘s github
Time consuming: 6.04s
Answer: GitHub is a web-based platform used for version control and collaboration of coding projects. It allows individuals and teams to store, share, and collaborate on changes to code, software, and applications. It also provides features such as issue tracking, project management tools, and code review. It is one of the most popular and widely used online platforms for open-source projects.

OpenAI API + GPTCache, exact match cache#

Initalize the cache to run GPTCache and import openai form gptcache.adapter, which will automatically set the map data manager to match the exact cahe, more details refer to build your cache.

And if you ask ChatGPT the exact same two questions, the answer to the second question will be obtained from the cache without requesting ChatGPT again.

import time


def response_text(openai_resp):
    return openai_resp['choices'][0]['message']['content']

print("Cache loading.....")

# To use GPTCache, that's all you need
# -------------------------------------------------
from gptcache import cache
from gptcache.adapter import openai

cache.init()
cache.set_openai_key()
# -------------------------------------------------

question = "what's github"
for _ in range(2):
    start_time = time.time()
    response = openai.ChatCompletion.create(
      model='gpt-3.5-turbo',
      messages=[
        {
            'role': 'user',
            'content': question
        }
      ],
    )
    print(f'Question: {question}')
    print("Time consuming: {:.2f}s".format(time.time() - start_time))
    print(f'Answer: {response_text(response)}\n')
Cache loading.....
Question: what's github
Time consuming: 6.88s
Answer: GitHub is a web-based platform that allows developers to store, share, and collaborate on programming projects. It is primarily used for version control, where developers can work on different features and changes of a project simultaneously without overwriting each other's work. GitHub also provides tools for issue tracking, code review, and project management. It is widely used in the open-source community and by software development teams in organizations of all sizes.

Question: what's github
Time consuming: 0.00s
Answer: GitHub is a web-based platform that allows developers to store, share, and collaborate on programming projects. It is primarily used for version control, where developers can work on different features and changes of a project simultaneously without overwriting each other's work. GitHub also provides tools for issue tracking, code review, and project management. It is widely used in the open-source community and by software development teams in organizations of all sizes.

OpenAI API + GPTCache, similar search cache#

Set the cache with embedding_func to generate embedding for the text, and data_manager to manager the cache data, similarity_evaluation to evaluate the similarities, more details refer to build your cache.

After obtaining an answer from ChatGPT in response to several similar questions, the answers to subsequent questions can be retrieved from the cache without the need to request ChatGPT again.

import time


def response_text(openai_resp):
    return openai_resp['choices'][0]['message']['content']

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

print("Cache loading.....")

onnx = Onnx()
data_manager = get_data_manager(CacheBase("sqlite"), VectorBase("faiss", dimension=onnx.dimension))
cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    )
cache.set_openai_key()

questions = [
    "what's github",
    "can you explain what GitHub is",
    "can you tell me more about GitHub",
    "what is the purpose of GitHub"
]

for question in questions:
    start_time = time.time()
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        messages=[
            {
                'role': 'user',
                'content': question
            }
        ],
    )
    print(f'Question: {question}')
    print("Time consuming: {:.2f}s".format(time.time() - start_time))
    print(f'Answer: {response_text(response)}\n')
Cache loading.....
Question: what's github
Time consuming: 7.11s
Answer: GitHub is a web-based platform that allows developers to store, manage, review, and collaborate on code repositories. It is a version control system that enables developers to track changes they make in code over time and collaborate on projects with other developers. GitHub is used by millions of developers worldwide to share code, collaborate on open-source projects, and contribute to projects owned by others. It's also a hub for various communities and forums related to software development.

Question: can you explain what GitHub is
Time consuming: 0.19s
Answer: GitHub is a web-based platform that allows developers to store, manage, review, and collaborate on code repositories. It is a version control system that enables developers to track changes they make in code over time and collaborate on projects with other developers. GitHub is used by millions of developers worldwide to share code, collaborate on open-source projects, and contribute to projects owned by others. It's also a hub for various communities and forums related to software development.

Question: can you tell me more about GitHub
Time consuming: 0.23s
Answer: GitHub is a web-based platform that allows developers to store, manage, review, and collaborate on code repositories. It is a version control system that enables developers to track changes they make in code over time and collaborate on projects with other developers. GitHub is used by millions of developers worldwide to share code, collaborate on open-source projects, and contribute to projects owned by others. It's also a hub for various communities and forums related to software development.

Question: what is the purpose of GitHub
Time consuming: 0.21s
Answer: GitHub is a web-based platform that allows developers to store, manage, review, and collaborate on code repositories. It is a version control system that enables developers to track changes they make in code over time and collaborate on projects with other developers. GitHub is used by millions of developers worldwide to share code, collaborate on open-source projects, and contribute to projects owned by others. It's also a hub for various communities and forums related to software development.