Summary Buffer Memory¶

The same CassandraChatMessageHistory can power a more sophisticated kind of chat memory that automatically maintains a summary of the past interactions, so as to provide a memory that virtually extends arbitrarily far back in the past while fitting a finite amount of tokens.

In [1]:

from langchain.memory import CassandraChatMessageHistory
from langchain.memory import ConversationSummaryBufferMemory
from langchain.chains import ConversationChain

from langchain.memory import CassandraChatMessageHistory from langchain.memory import ConversationSummaryBufferMemory from langchain.chains import ConversationChain

In [2]:

from cqlsession import getCQLSession, getCQLKeyspace
cqlMode = 'astra_db' # 'astra_db'/'local'
session = getCQLSession(mode=cqlMode)
keyspace = getCQLKeyspace(mode=cqlMode)

from cqlsession import getCQLSession, getCQLKeyspace cqlMode = 'astra_db' # 'astra_db'/'local' session = getCQLSession(mode=cqlMode) keyspace = getCQLKeyspace(mode=cqlMode)

In [3]:

message_history = CassandraChatMessageHistory(
    session_id='summarized-conversation-4321',
    session=session,
    keyspace=keyspace,
    ttl_seconds = 3600,
)

message_history = CassandraChatMessageHistory( session_id='summarized-conversation-4321', session=session, keyspace=keyspace, ttl_seconds = 3600, )

In [4]:

message_history.clear()

message_history.clear()

Below is the logic to instantiate the LLM of choice. We choose to leave it in the notebooks for clarity.

In [5]:

from llm_choice import suggestLLMProvider

llmProvider = suggestLLMProvider()
# (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI' ... manually if you have credentials)

if llmProvider == 'GCP_VertexAI':
    from langchain.llms import VertexAI
    llm = VertexAI()
    print('LLM from VertexAI')
elif llmProvider == 'OpenAI':
    from langchain.llms import OpenAI
    llm = OpenAI()
    print('LLM from OpenAI')
else:
    raise ValueError('Unknown LLM provider.')

from llm_choice import suggestLLMProvider llmProvider = suggestLLMProvider() # (Alternatively set llmProvider to 'GCP_VertexAI', 'OpenAI' ... manually if you have credentials) if llmProvider == 'GCP_VertexAI': from langchain.llms import VertexAI llm = VertexAI() print('LLM from VertexAI') elif llmProvider == 'OpenAI': from langchain.llms import OpenAI llm = OpenAI() print('LLM from OpenAI') else: raise ValueError('Unknown LLM provider.')

LLM from OpenAI

Use the memory with summary in a Chain¶

In [6]:

memory = ConversationSummaryBufferMemory(
    llm=llm,
    chat_memory=message_history,
    max_token_limit=180,
)

memory = ConversationSummaryBufferMemory( llm=llm, chat_memory=message_history, max_token_limit=180, )

In [7]:

summaryConversation = ConversationChain(
    llm=llm, 
    memory=memory,
    verbose=True
)

summaryConversation = ConversationChain( llm=llm, memory=memory, verbose=True )

With the verbose output you can track the summary. It starts empty and once few messages have been exchanged it starts to "incorporate" those messages that would have been lost in the past.

In [9]:

summaryConversation.predict(input='Tell me about William Tell, the hero')

summaryConversation.predict(input='Tell me about William Tell, the hero')

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a folk hero from Switzerland who is best known for his legend of shooting an apple off his son's head. He was a master marksman with a crossbow and is said to have used it to shoot down an apple placed on his son's head. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:

> Finished chain.

Out[9]:

" William Tell is a legendary Swiss folk hero who is most famous for a legend involving him shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills. He is also known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity."

In [10]:

summaryConversation.predict(
    input="Any relation to the apple from Cupertino?"
)

summaryConversation.predict( input="Any relation to the apple from Cupertino?" )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
Human: Tell me about William Tell, the hero
AI:  William Tell is a folk hero from Switzerland who is best known for his legend of shooting an apple off his son's head. He was a master marksman with a crossbow and is said to have used it to shoot down an apple placed on his son's head. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero who is most famous for a legend involving him shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills. He is also known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity.
Human: Any relation to the apple from Cupertino?
AI:

> Finished chain.

Out[10]:

' No, there is no relation between William Tell and the apple from Cupertino. The apple in the William Tell legend is a literal apple, while the apple from Cupertino is a technology company.'

No messages have slided past the "recent window" yet -- correspondingly, the summary buffer is still empty:

In [11]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[11]:

"\nThe human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The AI then goes on to tell the human about William Tell, a folk hero from Switzerland best known for his legend of shooting an apple off his son's head. He is also known as the symbol of Swiss freedom and independence."

Continue the conversation - you might start to see the summary of past exchanges making its way to the final prompt for the LLM in the form of a "System" contribution to the Current conversation:

In [12]:

summaryConversation.predict(
    input="I heard the two apples may be in fact the very same fruit."
)

summaryConversation.predict( input="I heard the two apples may be in fact the very same fruit." )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The AI then goes on to tell the human about William Tell, a folk hero from Switzerland best known for his legend of shooting an apple off his son's head. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:  William Tell is a folk hero from Switzerland who is best known for his legend of shooting an apple off his son's head. He was a master marksman with a crossbow and is said to have used it to shoot down an apple placed on his son's head. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero who is most famous for a legend involving him shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills. He is also known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity.
Human: Any relation to the apple from Cupertino?
AI:  No, there is no relation between William Tell and the apple from Cupertino. The apple in the William Tell legend is a literal apple, while the apple from Cupertino is a technology company.
Human: I heard the two apples may be in fact the very same fruit.
AI:

> Finished chain.

Out[12]:

" I don't know the answer to that question."

This is where the current summary buffer is stored

(Note: depending on the length of the messages so far, at this point you might or might not see an empty summary. In any case, keep reading and it will doubtless start to be nonempty.)

In [13]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[13]:

"\nThe human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The AI then goes on to explain that William Tell is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is also known as the symbol of Swiss freedom and independence."

The whole conversation is still in the "recent" window. Below is what will start to happen in the ConversationSummaryBufferMemory at the next chat message: starting from the message list, and the summary so far (if any), the LLM is asked to come up with a new summary which will overwrite the current one, stored in memory.moving_summary_buffer.

In [14]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The AI then goes on to explain that William Tell is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is also known as the symbol of Swiss freedom and independence. The human then asks for more information about William Tell, to which the AI responds that he was a master marksman with a crossbow and is said to have used it to shoot down an apple placed on his son's head. The human then asked if there is any relation between William Tell's apple and the apple from Cupertino, to which the AI responded that there is no relation between them. The human then suggested that the two apples may be the same fruit, to which the AI responded that it doesn't know the answer to that question.

In [15]:

summaryConversation.predict(
    input='Ok, enough with apples. Let\'s talk starfish now.'
)

summaryConversation.predict( input='Ok, enough with apples. Let\'s talk starfish now.' )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. The AI then goes on to explain that William Tell is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:  William Tell is a folk hero from Switzerland who is best known for his legend of shooting an apple off his son's head. He was a master marksman with a crossbow and is said to have used it to shoot down an apple placed on his son's head. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero who is most famous for a legend involving him shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills. He is also known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity.
Human: Any relation to the apple from Cupertino?
AI:  No, there is no relation between William Tell and the apple from Cupertino. The apple in the William Tell legend is a literal apple, while the apple from Cupertino is a technology company.
Human: I heard the two apples may be in fact the very same fruit.
AI:  I don't know the answer to that question.
Human: Ok, enough with apples. Let's talk starfish now.
AI:

> Finished chain.

Out[15]:

' Sure! Starfish are a type of marine invertebrate, also known as sea stars. They have a unique body shape, often with five or more arms, and they are found in a variety of ocean habitats around the world. They have evolved to survive in harsh environments and are capable of regenerating body parts if they are damaged.'

In [16]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[16]:

"\nThe human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. When the human asks about William Tell, the AI explains that he is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is also said to have done this to demonstrate his archery skills and is known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity."

Now the chat history has presumably reached a certain threshold and there is a summary. In the verbose output below, you will see it appearing in the prompt as a "System" role in the conversation recap.

In [17]:

summaryConversation.predict(
    input='I hear that some of those destroy marine ecosystems.'
)

summaryConversation.predict( input='I hear that some of those destroy marine ecosystems.' )

> Entering new ConversationChain chain...
Prompt after formatting:
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
System: 
The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. When the human asks about William Tell, the AI explains that he is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is also said to have done this to demonstrate his archery skills and is known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity.
Human: Tell me about William Tell, the hero
AI:  William Tell is a folk hero from Switzerland who is best known for his legend of shooting an apple off his son's head. He was a master marksman with a crossbow and is said to have used it to shoot down an apple placed on his son's head. He is also known as the symbol of Swiss freedom and independence.
Human: Tell me about William Tell, the hero
AI:  William Tell is a legendary Swiss folk hero who is most famous for a legend involving him shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills. He is also known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity.
Human: Any relation to the apple from Cupertino?
AI:  No, there is no relation between William Tell and the apple from Cupertino. The apple in the William Tell legend is a literal apple, while the apple from Cupertino is a technology company.
Human: I heard the two apples may be in fact the very same fruit.
AI:  I don't know the answer to that question.
Human: Ok, enough with apples. Let's talk starfish now.
AI:  Sure! Starfish are a type of marine invertebrate, also known as sea stars. They have a unique body shape, often with five or more arms, and they are found in a variety of ocean habitats around the world. They have evolved to survive in harsh environments and are capable of regenerating body parts if they are damaged.
Human: I hear that some of those destroy marine ecosystems.
AI:

> Finished chain.

Out[17]:

' Yes, certain species of starfish can have a negative impact on their environment. For example, the crown-of-thorns starfish is a species of starfish native to the Indo-Pacific region that feeds on coral polyps. Overpopulation of this species can lead to coral reef destruction.'

In [18]:

memory.moving_summary_buffer

memory.moving_summary_buffer

Out[18]:

"\nThe human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. When the human asks about William Tell, the AI explains that he is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills and is known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity. When asked if there is any relation between William Tell and the apple from Cupertino, the AI explains that there is no relation between the two apples, as the apple in the William Tell legend is a literal apple, while the apple from Cupertino is a technology company."

From this point on, the summary will not grow indefinitely in size, but will rather try to cover a longer and longer conversation in a fixed amount of text.

In [19]:

print(memory.predict_new_summary(
    memory.chat_memory.messages,
    memory.moving_summary_buffer,
))

print(memory.predict_new_summary( memory.chat_memory.messages, memory.moving_summary_buffer, ))

The human asks what the AI thinks of artificial intelligence. The AI thinks artificial intelligence is a force for good because it will help humans reach their full potential. When the human asks about William Tell, the AI explains that he is a folk hero from Switzerland best known for his legend of shooting an apple off his son's head with a crossbow. He is said to have done this to demonstrate his archery skills and is known as the symbol of Swiss freedom and independence, and is part of the Swiss national identity. When asked if there is any relation between William Tell and the apple from Cupertino, the AI explains that there is no relation between the two apples, as the apple in the William Tell legend is a literal apple, while the apple from Cupertino is a technology company. When the conversation turns to starfish, the AI explains that starfish are a type of marine invertebrate, also known as sea stars, which have a unique body shape and are capable of regenerating body parts if they are damaged. However, some species of starfish can have a negative impact on their environment, like the crown-of-thorns starfish which feeds on coral polyps and can lead to coral reef destruction if overpopulated.