Processor

Index

• Processor

  • post

  • check_hit

  • summarization_context

  • selective_context

  • context

  • concat_context

  • pre

post

gptcache.processor.post.random_one(messages: List[Any]) → Any

Randomly select one result after evaluation.

Parameters

messages (List[Any]) – A list of candidate outputs.

Example

from gptcache.processor.post import random_one

messages = ["message 1", "message 2", "message 3"]
answer = random_one(messages)

gptcache.processor.post.first(messages: List[Any]) → Any

Get the first result after evaluation.

Parameters

messages (List[Any]) – A list of candidate outputs.

Example

from gptcache.processor.post import first

messages = ["message 1", "message 2", "message 3"]
answer = first(messages)
assert answer = messages[0]

gptcache.processor.post.nop(messages: List[Any]) → Any

No change after evaluation.

Parameters

messages (List[Any]) – A list of candidate outputs.

Example

from gptcache.processor.post import nop

messages = ["message 1", "message 2", "message 3"]
answer = nop(messages)
assert answer = messages

gptcache.processor.post.temperature_softmax(messages: List[Any], scores: List[float], temperature: float = 0.0) → Any

Post processing with temperature softmax after evaluation.

Parameters

• messages (List[Any]) – A list of candidate outputs.

• scores (List[float]) – A list of evaluation scores corresponding to messages

• temperature (float) – A non-negative number of sampling temperature, defaults to 0. A higher temperature makes the output more random. A lower temperature means a more deterministic and confident output.

Example

from gptcache.processor.post import temperature_softmax

messages = ["message 1", "message 2", "message 3"]
scores = [0.9, 0.5, 0.1]
answer = temperature_softmax(messages, scores, temperature=0.5)

check_hit

gptcache.processor.check_hit.check_hit_session(cur_session_id: str, cache_session_ids: list, cache_questions: list, cache_answer: str)

Check if the sesion result meets the hit requirement.

Parameters

• cur_session_id (str) – the name of the current session.

• cache_session_ids (list) – a list of session names for caching the same content if you are using map as a data management method. Otherwise a list of session names for similar content and same answer.

• cache_question (list) – a list with one question which same as the you asked if you use a map as a data management method. Otherwise it is a list that is similar to the question you asked with the same answer, and it is correspondence with cache_session_ids.

• cache_answer – the content of the cached answer.

• cache_answer – str

Returns

True or False

summarization_context

gptcache.processor.context.summarization_context.summarize_to_length(summarizer, text, target_len, max_len=1024)

class gptcache.processor.context.summarization_context.SummarizationContextProcess(model_name='facebook/bart-large-cnn', tokenizer=None, target_length=512)

Bases: gptcache.processor.context.context.ContextProcess

A context processor for summarizing large amounts of text data using a summarizer model.

Parameters

• summarizer (transformers.PreTrainedModel) – The summarizer model to use for summarization.

• tokenizer – The tokenizer to use for tokenizing the text data.

It used for measuring the output length. :type tokenizer: transformers.PreTrainedTokenizer :param target_length: The length of the summarized text. :type target_length: int

Example

from gptcache.processor.context.summarization_context import SummarizationContextProcess

context_process = SummarizationContextProcess()
cache.init(pre_embedding_func=context_process.pre_process)

summarize_to_sentence(sentences, target_size=1000)

format_all_content(data: Dict[str, Any], **params: Dict[str, Any])

format all content of the llm request data as a string

Parameters

data (Dict[str, Any]) – the user llm request data

process_all_content()

process all content of the llm request data, for extracting key information in context. In order to achieve this goal, you can pass the summary model and so on

selective_context

class gptcache.processor.context.selective_context.SelectiveContextProcess(model_type: str = 'gpt2', lang: str = 'en', reduce_ratio: float = 0.35, reduce_level: str = 'phrase')

Bases: gptcache.processor.context.context.ContextProcess

A context processor for selecting context

Need to download the corresponding model before use, the default English model is: en_core_web_sm

pip install spacy && python -m spacy download en_core_web_sm

Parameters

• model_type (str) – the selective context model name, default value is ‘gpt2’

• lang (str) – the content lang type, default value is ‘en’.

• reduce_ratio (float) – selective context ratio. The range for the value is between 0 and 1, with a default value of 0.35.

• reduce_level (str) – selective context level. The valid values include ‘sent’, ‘phrase’, and ‘token’, with the default value being ‘phrase’.

more details: liyucheng09/Selective_Context

Example

from gptcache.processor.context.selective_context import SelectiveContextProcess

context_process = SelectiveContextProcess()
cache.init(pre_embedding_func=context_process.pre_process)

content: str = ''

format_all_content(data: Dict[str, Any], **params: Dict[str, Any])

format all content of the llm request data as a string

Parameters

data (Dict[str, Any]) – the user llm request data

process_all_content()

process all content of the llm request data, for extracting key information in context. In order to achieve this goal, you can pass the summary model and so on

context

class gptcache.processor.context.context.ContextProcess

Bases: object

ContextProcess: the context process interfacer, which is used to pre-process the lang conversation. By the way, the GPTCache will acquire more information and get a more accurate embedding vector.

Example

from gptcache.processor.context import SummarizationContextProcess

context_process = SummarizationContextProcess()
cache.init(pre_embedding_func=context_process.pre_process)

abstract format_all_content(data: Dict[str, Any], **params: Dict[str, Any])

format all content of the llm request data as a string

Parameters

data (Dict[str, Any]) – the user llm request data

abstract process_all_content()

process all content of the llm request data, for extracting key information in context. In order to achieve this goal, you can pass the summary model and so on

pre_process(data: Dict[str, Any], **params: Dict[str, Any])

pre-process function, it’s used as the GPTCache initialization param – pre_embedding_func.

Parameters

data (Dict[str, Any]) – the user llm request data

concat_context

class gptcache.processor.context.concat_context.ConcatContextProcess

Bases: gptcache.processor.context.context.ContextProcess

A concat context processor simply concat the context. Generally used with rwkv embedding, because rwkv can input almost infinitely long

Example

from gptcache.manager import manager_factory
from gptcache.processor.context.concat_context import ConcatContextProcess

context_process = ConcatContextProcess()
rwkv_embedding = Rwkv()
data_manager = manager_factory(
    "sqlite,faiss",
    vector_params={"dimension": rwkv_embedding.dimension},
)
cache.init(
    pre_embedding_func=context_process.pre_process,
    embedding_func=rwkv_embedding.to_embeddings,
    data_manager=data_manager,
)

content: str = ''

format_all_content(data: Dict[str, Any], **params: Dict[str, Any])

format all content of the llm request data as a string

Parameters

data (Dict[str, Any]) – the user llm request data

process_all_content()

process all content of the llm request data, for extracting key information in context. In order to achieve this goal, you can pass the summary model and so on

pre

gptcache.processor.pre.last_content(data: Dict[str, Any], **_: Dict[str, Any]) → Any

get the last content of the message list

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import last_content

content = last_content({"messages": [{"content": "foo1"}, {"content": "foo2"}]})
# content = "foo2"

gptcache.processor.pre.last_content_without_prompt(data: Dict[str, Any], **params: Dict[str, Any]) → Any

get the last content of the message list without prompts content

Parameters

• data (Dict[str, Any]) – the user llm request data

• params (Dict[str, Any]) – the special gptcache params, like prompts param in the cache object

Example

from gptcache.processor.pre import last_content_without_prompt

content = last_content_without_prompt(
        {"messages": [{"content": "foo1"}, {"content": "foo2"}]}, prompts=["foo"]
    )
# content = "2"

gptcache.processor.pre.last_content_without_template(data: Dict[str, Any], **params: Dict[str, Any]) → Any

get the last content’s template values of the message list without template content.

When considering a cache agent or chain, the majority of the content consists of template content, while the essential information is simply a list of parameters within the template. In this way, the cache key is composed of a string made up of all the parameter values in the list.

WARNING: Two parameters without intervals cannot appear in the template, for example: template = “{foo}{hoo}” is not supported, but template = “{foo}:{hoo}” is supported

Parameters

data (Dict[str, Any]) – the user llm request data

Example with str template

from gptcache import Config
from gptcache.processor.pre import last_content_without_template

template_obj = "tell me a joke about {subject}"
prompt = template_obj.format(subject="animal")
value = last_content_without_template(
    data={"messages": [{"content": prompt}]}, cache_config=Config(template=template_obj)
)
print(value)
# ['animal']

Example with langchain template

from langchain import PromptTemplate

from gptcache import Config
from gptcache.processor.pre import last_content_without_template

template_obj = PromptTemplate.from_template("tell me a joke about {subject}")
prompt = template_obj.format(subject="animal")

value = last_content_without_template(
    data={"messages": [{"content": prompt}]},
    cache_config=Config(template=template_obj.template),
)
print(value)
# ['animal']

NOTE: At present, only the simple PromptTemplate in langchain is supported. For ChatPromptTemplate, it needs to be adjusted according to the template array. If you need to use it, you need to pass in the final dialog template yourself. The reason why it cannot be advanced is that ChatPromptTemplate does not provide a method to directly return the template string.

gptcache.processor.pre.all_content(data: Dict[str, Any], **_: Dict[str, Any]) → Any

get all content of the message list

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import all_content

content = all_content(
    {"messages": [{"content": "foo1"}, {"content": "foo2"}]}
)
# content = "foo1\nfoo2"

gptcache.processor.pre.nop(data: Dict[str, Any], **_: Dict[str, Any]) → Any

do nothing of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import nop

content = nop({"str": "hello"})
# {"str": "hello"}

gptcache.processor.pre.get_prompt(data: Dict[str, Any], **_: Dict[str, Any]) → Any

get the prompt of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_prompt

content = get_prompt({"prompt": "foo"})
# "foo"

gptcache.processor.pre.get_file_name(data: Dict[str, Any], **_: Dict[str, Any]) → str

get the file name of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_file_name

file = open("test.txt", "a")
content = get_file_name({"file": file})
# "test.txt"

gptcache.processor.pre.get_file_bytes(data: Dict[str, Any], **_: Dict[str, Any]) → bytes

get the file bytes of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_file_bytes

content = get_file_bytes({"file": open("test.txt", "rb")})

gptcache.processor.pre.get_input_str(data: Dict[str, Any], **_: Dict[str, Any]) → str

get the image and question str of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_input_str

content = get_input_str({"input": {"image": open("test.png", "rb"), "question": "foo"}})

gptcache.processor.pre.get_input_image_file_name(data: Dict[str, Any], **_: Dict[str, Any]) → str

get the image file name of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_input_image_file_name

content = get_input_image_file_name({"input": {"image": open("test.png", "rb")}})
# "test.png"

gptcache.processor.pre.get_image_question(data: Dict[str, Any], **_: Dict[str, Any]) → str

get the image and question str of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_image_question

content = get_image_question({"image": open("test.png", "rb"), "question": "foo"})

gptcache.processor.pre.get_image(data: Dict[str, Any], **_: Dict[str, Any]) → str

get the image of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_image

content = get_image({"image": open("test.png", "rb")})
# "test.png"

gptcache.processor.pre.get_inputs(data: Dict[str, Any], **_: Dict[str, Any])

get the inputs of the llm request params

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_inputs

content = get_inputs({"inputs": "hello"})
# "hello"

gptcache.processor.pre.get_messages_last_content(data: Dict[str, Any], **_: Any) → str

get the last content of the llm request messages array

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import get_messages_last_content

content = get_messages_last_content({"messages": [{"content": "hello"}, {"content": "world"}]})
# "world"

gptcache.processor.pre.get_openai_moderation_input(data: Dict[str, Any], **_: Dict[str, Any]) → str

get the input param of the openai moderation request params

Parameters

data (Dict[str, Any]) – the user openai moderation request data

Example

from gptcache.processor.pre import get_openai_moderation_input

content = get_openai_moderation_input({"input": ["hello", "world"]})
# "['hello', 'world']"

gptcache.processor.pre.concat_all_queries(data: Dict[str, Any], **params: Dict[str, Any]) → Any

Parameters

data (Dict[str, Any]) – the user llm request data

Example

from gptcache.processor.pre import concat_all_queries

content = concat_all_queries({"messages": [{"role": "system", "content": "hello"},
    {"role": "user", "content": "world"},
    {"role": "assistant", "content": "alice"}]})