OpenAI Chat with Temperature#

Temperature in deep learning is a parameter usually used to adjust the probability distribution of the predicted outputs. It is also known as softmax temperature or softmax scaling. In simple terms, it controls the level of confidence that a neural network has in its predictions. It helps to increase the diversity of the model’s outputs.

For temperature in OpenAI chat request, “higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic” as explained in OpenAI documentation.

GPTCache also enables a similar temperature parameter at request, in range of [0.0, 2.0], which works at two stages:

control the possibility of sending request to OpenAI directly without searching in cache
affect the selection of final answer from potential answers retrieved from cache

Let us try the adapted OpenAI Chat API with GPTCache enabled and see how temperature affects output given the same question.

Set Cache#

Initiate GPTCache with preferred configurations and modules.

import time

from gptcache import cache, Config
from gptcache.manager import manager_factory
from gptcache.embedding import Onnx
from gptcache.processor.post import temperature_softmax
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
from gptcache.adapter import openai


onnx = Onnx()
data_manager = manager_factory("sqlite,faiss", vector_params={"dimension": onnx.dimension})

cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    post_process_messages_func=temperature_softmax
    )
# cache.config = Config(similarity_threshold=0.2)

Getting Started#

cache.set_openai_key()
question = 'what is github'

default: temperature=0.0#

If temperature is not specified in request, it will use the default value 0. When temperature is 0, it will firstly search through cache and return the most confident answer retrieved from cache. If there is no satisfactory answer available in cache, it will continue on sending request to OpenAI.

for _ in range(3):
    # use cache without temperature (temperature=0.0)
    start = time.time()
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        messages=[{
            'role': 'user',
            'content': question
        }],
    )
    print('Time elapsed:', round(time.time() - start, 3))
    print(response["choices"][0]["message"]["content"])

Time elapsed: 7.906
GitHub is a web-based platform that is used to manage, store, and share software development projects. It offers a version control system and collaboration tools for developers to work together on code and other digital assets. GitHub is popular in the open-source community, but it is also used by companies to manage their proprietary code. It allows developers to easily contribute to projects, track changes, and manage project workflows. It also provides tools for issue tracking, documentation, and continuous integration and deployment.
Time elapsed: 0.22
GitHub is a web-based platform that is used to manage, store, and share software development projects. It offers a version control system and collaboration tools for developers to work together on code and other digital assets. GitHub is popular in the open-source community, but it is also used by companies to manage their proprietary code. It allows developers to easily contribute to projects, track changes, and manage project workflows. It also provides tools for issue tracking, documentation, and continuous integration and deployment.
Time elapsed: 0.239
GitHub is a web-based platform that is used to manage, store, and share software development projects. It offers a version control system and collaboration tools for developers to work together on code and other digital assets. GitHub is popular in the open-source community, but it is also used by companies to manage their proprietary code. It allows developers to easily contribute to projects, track changes, and manage project workflows. It also provides tools for issue tracking, documentation, and continuous integration and deployment.

maximum: temperature=2.0#

When temperature is at its maximum value of 2, it will skip searching cache and send request to OpenAI directly.

# use cache with temperature 2.0
for _ in range(3):
    start = time.time()
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        temperature=2.0,
        max_tokens=30,
        messages=[{
            'role': 'user',
            'content': question
        }],
    )
    print('Time elapsed:', round(time.time() - start, 3))
    print(response["choices"][0]["message"]["content"])

Time elapsed: 2.675
GitHub is a web-based platform used for version control and collaboration that helps developers store and manage their code repositories online. It allows multiple developers to work collabor
Time elapsed: 2.667
GitHub is a web-based platform used for version control and collaboration in software development projects. It provides a centralized location for developers to manage and store their code
Time elapsed: 2.56
GitHub is a web-based platform where developers can store, share, and collaborate on their code projects. It is also a version control system, meaning it

0.0<temperature<2.0#

When temperature is between 0 and 2, a higher value will increase the probability of skipping cache search and makes the output more random.

# use cache with temperature 1.0
for _ in range(3):
    start = time.time()
    response = openai.ChatCompletion.create(
        model='gpt-3.5-turbo',
        temperature=1.0,
        messages=[{
            'role': 'user',
            'content': question
        }],
    )
    print('Time elapsed:', round(time.time() - start, 3))
    print(response["choices"][0]["message"]["content"])

Time elapsed: 0.197
GitHub is a web-based platform used for version control and collaboration that helps developers store and manage their code repositories online. It allows multiple developers to work collabor
Time elapsed: 6.116
GitHub is a web-based platform that hosts and manages software development projects using the Git version control system. It provides a collaborative environment for developers to work together on coding projects, including features such as task management, code review, and bug tracking. GitHub enables developers to share their code with the rest of the community, discover new projects and contribute to them, collaborate with others on open-source software, and showcase their work to potential employers.
Time elapsed: 6.757
GitHub is a web-based platform used for version control and collaboration of software development projects. It provides tools for developers to manage and store their code, as well as to collaborate with others through features such as pull requests, code reviews, and issue tracking. GitHub has become a popular platform for open-source projects and offers various features such as version control, documentation, bug tracking, task management, wikis, and more. It is widely used in the technology industry and by developers all over the world.

OpenAI Chat with Temperature

Contents