哈嘍！伙伴們，我是小智，你們的AI向?qū)?。歡迎來(lái)到每日的AI學(xué)習(xí)時(shí)間。今天，我們將一起深入AI的奇妙世界，探索“面向開(kāi)發(fā)者的LLM入門(mén)教程-檢索(Retrieval)英文版”，并學(xué)會(huì)本篇文章中所講的全部知識(shí)點(diǎn)。還是那句話(huà)“不必遠(yuǎn)征未知，只需喚醒你的潛能！”跟著小智的步伐，我們終將學(xué)有所成，學(xué)以致用，并發(fā)現(xiàn)自身的更多可能性。話(huà)不多說(shuō)，現(xiàn)在就讓我們開(kāi)始這場(chǎng)激發(fā)潛能的AI學(xué)習(xí)之旅吧。

面向開(kāi)發(fā)者的LLM入門(mén)教程-檢索(Retrieval)英文版：

英文版

1.相似性檢索

from langchain.vectorstores import Chroma
from langchain.embeddings.openai import OpenAIEmbeddings
persist_directory = ‘docs/chroma/cs229_lectures/’
embedding = OpenAIEmbeddings()
vectordb = Chroma(
persist_directory=persist_directory,
embedding_function=embedding
)
print(vectordb._collection.count())

209

簡(jiǎn)單示例

texts = [
“””The Amanita phalloides has a large and imposing epigeous (aboveground)
fruiting body (basidiocarp).”””,
“””A mushroom with a large fruiting body is the Amanita phalloides. Some
varieties are all-white.”””,
“””A. phalloides, a.k.a Death Cap, is one of the most poisonous of all known
mushrooms.”””,
]
smalldb = Chroma.from_texts(texts, embedding=embedding)
question = “Tell me about all-white mushrooms with large fruiting bodies”
print(“相似性檢索：”)
print(smalldb.similarity_search(question, k=2))
print(“MMR 檢索：”)
print(smalldb_chinese.max_marginal_relevance_search(question,k=2, fetch_k=3))

0%| | 0/1 [00:00

相似性檢索：
[Document(page_content=’A mushroom with a large fruiting body is the Amanita
phalloides. Some varieties are all-white.’, metadata={}),
Document(page_content=’The Amanita phalloides has a large and imposing epigeous
(aboveground) fruiting body (basidiocarp).’, metadata={})]
MMR 檢索：
[Document(page_content=’一種具有大型子實(shí)體的蘑菇是毒鵝膏菌（Amanita phalloides）。某些品
種全白。’, metadata={}), Document(page_content=’A. phalloides，又名死亡帽，是已知所有蘑
菇中最有毒的一種。’, metadata={})]

2.最大邊際相關(guān)性

question = “what did they say about matlab?”
docs_ss = vectordb.similarity_search(question,k=3)
print(“相似性檢索：”)
print(“docs[0]: “)
print(docs_ss[0].page_content[:100])
print()
print(“docs[1]: “)
print(docs_ss[1].page_content[:100])
print()

docs_mmr = vectordb.max_marginal_relevance_search(question,k=3)
print(“MMR 檢索：”)
print(“mmr[0]: “)
print(docs_mmr[0].page_content[:100])
print()
print(“MMR 檢索：”)
print(“mmr[1]: “)
print(docs_mmr[1].page_content[:100])

相似性檢索：
docs[0]:
those homeworks will be done in either MATLA B or in Octave, which is sort of — I
know some people

docs[1]:
those homeworks will be done in either MATLA B or in Octave, which is sort of — I
know some people

MMR 檢索：
mmr[0]:
those homeworks will be done in either MATLA B or in Octave, which is sort of — I
know some people

MMR 檢索：
mmr[1]:
algorithm then? So what’s different? How come I was making all that noise
earlier about
least squa

3.使用元數(shù)據(jù)

question = “what did they say about regression in the third lecture?”

docs = vectordb.similarity_search(
question,
k=3,
filter={“source”:”docs/cs229_lectures/MachineLearning-Lecture03.pdf”}
)

for d in docs:
print(d.metadata)

{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 0}
{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 14}
{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 4}

4.使用自查詢(xún)檢索器

from langchain.llms import OpenAI
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chains.query_constructor.base import AttributeInfo
llm = OpenAI(temperature=0)
metadata_field_info = [
AttributeInfo(
name=”source”,
description=”The lecture the chunk is from, should be one of
`docs/cs229_lectures/MachineLearning-Lecture01.pdf`,
`docs/cs229_lectures/MachineLearning-Lecture02.pdf`, or
`docs/cs229_lectures/MachineLearning-Lecture03.pdf`”,
type=”string”,
),
AttributeInfo(
name=”page”,
description=”The page from the lecture”,
type=”integer”,
),
]
document_content_description = “Lecture notes”
retriever = SelfQueryRetriever.from_llm(
llm,
vectordb,
document_content_description,
metadata_field_info,
verbose=True
)
question = “what did they say about regression in the third lecture?”
docs = retriever.get_relevant_documents(question)
for d in docs:
print(d.metadata)

/root/autodl-tmp/env/gpt/lib/python3.10/sitepackages/langchain/chains/llm.py:275: UserWarning: The predict_and_parse method
is deprecated, instead pass an output parser directly to LLMChain.
warnings.warn(

query=’regression’ filter=Comparison(comparator=,
attribute=’source’, value=’docs/cs229_lectures/MachineLearning-Lecture03.pdf’)
limit=None
{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 14}
{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 0}
{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 10}
{‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture03.pdf’, ‘page’: 10}

5.壓縮

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor
def pretty_print_docs(docs):
print(f”n{‘-‘ * 100}n”.join([f”Document {i+1}:nn” + d.page_content for i,
d in enumerate(docs)]))
llm = OpenAI(temperature=0)
compressor = LLMChainExtractor.from_llm(llm) # 壓縮器
compression_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=vectordb.as_retriever()
)
question = “what did they say about matlab?”
compressed_docs = compression_retriever.get_relevant_documents(question)
pretty_print_docs(compressed_docs)

Document 1:
“MATLAB is I guess part of the programming language that makes it very easy to
write codes using matrices, to write code for numerical routines, to move data
around, to plot data. And it’s sort of an extremely easy to learn tool to use for
implementing a lot of learning algorithms.”
———————————————————————————
——————-
Document 2:
“MATLAB is I guess part of the programming language that makes it very easy to
write codes using matrices, to write code for numerical routines, to move data
around, to plot data. And it’s sort of an extremely easy to learn tool to use for
implementing a lot of learning algorithms.”
———————————————————————————
——————-
Document 3:
“And the student said, “Oh, it was the MATLAB.” So for those of you that don’t
know MATLAB yet, I hope you do learn it. It’s not hard, and we’ll actually have a
short MATLAB tutorial in one of the discussion sections for those of you that
don’t know it.”
———————————————————————————
——————-
Document 4:
“And the student said, “Oh, it was the MATLAB.” So for those of you that don’t
know MATLAB yet, I hope you do learn it. It’s not hard, and we’ll actually have a
short MATLAB tutorial in one of the discussion sections for those of you that
don’t know it.”

6.結(jié)合各種技術(shù)

compression_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=vectordb.as_retriever(search_type = “mmr”)
)
question = “what did they say about matlab?”
compressed_docs = compression_retriever.get_relevant_documents(question)
pretty_print_docs(compressed_docs)

Document 1:
“MATLAB is I guess part of the programming language that makes it very easy to
write codes using matrices, to write code for numerical routines, to move data
around, to plot data. And it’s sort of an extremely easy to learn tool to use for
implementing a lot of learning algorithms.”
———————————————————————————
——————-
Document 2:
“And the student said, “Oh, it was the MATLAB.” So for those of you that don’t
know MATLAB yet, I hope you do learn it. It’s not hard, and we’ll actually have a
short MATLAB tutorial in one of the discussion sections for those of you that
don’t know it.”

7.其他類(lèi)型的檢索

from langchain.retrievers import SVMRetriever
from langchain.retrievers import TFIDFRetriever
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
# 加載PDF
loader = PyPDFLoader(“docs/cs229_lectures/MachineLearning-Lecture01.pdf”)
pages = loader.load()
all_page_text = [p.page_content for p in pages]
joined_page_text = ” “.join(all_page_text)
# 分割文本
text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1500,chunk_overlap =
150)
splits = text_splitter.split_text(joined_page_text)
# 檢索
svm_retriever = SVMRetriever.from_texts(splits, embedding)
tfidf_retriever = TFIDFRetriever.from_texts(splits)
question = “What are major topics for this class?” # 這門(mén)課的主要主題是什么？
print(“SVM:”)
docs_svm = svm_retriever.get_relevant_documents(question)
print(docs_svm[0])
question = “what did they say about matlab?”
print(“TF-IDF:”)
docs_tfidf = tfidf_retriever.get_relevant_documents(question)
print(docs_tfidf[0])

SVM:
page_content=”let me just check what questions you have righ t now. So if there
are no questions, I’ll just nclose with two reminders, which are after class
today or as you start to talk with other npeople in this class, I just encourage
you again to start to form project partners, to try to nfind project partners to
do your project with. And also, this is a good time to start forming nstudy
groups, so either talk to your friends or post in the newsgroup, but we just
nencourage you to try to star t to do both of those today, okay? Form study
groups, and try nto find two other project partners. nSo thank you. I’m
looking forward to teaching this class, and I’ll see you in a couple of ndays.
[End of Audio] nDuration: 69 minutes” metadata={}
TF-IDF:
page_content=”Saxena and Min Sun here did, wh ich is given an image like this,
right? This is actually a npicture taken of the Stanford campus. You can apply
that sort of cl ustering algorithm and ngroup the picture into regions. Let me
actually blow that up so that you can see it more nclearly. Okay. So in the
middle, you see the lines sort of groupi ng the image together, ngrouping the
image into [inaudible] regions. nAnd what Ashutosh and Min did was they then
applied the learning algorithm to say can nwe take this clustering and us e it
to build a 3D model of the world? And so using the nclustering, they then had a
lear ning algorithm try to learn what the 3D structure of the nworld looks like
so that they could come up with a 3D model that you can sort of fly nthrough,
okay? Although many people used to th ink it’s not possible to take a single
nimage and build a 3D model, but using a lear ning algorithm and that sort of
clustering nalgorithm is the first step. They were able to. nI’ll just show
you one more example. I like this because it’s a picture of Stanford with our
nbeautiful Stanford campus. So again, taking th e same sort of clustering
algorithms, taking nthe same sort of unsupervised learning algor ithm, you can
group the pixels into different nregions. And using that as a pre-processing
step, they eventually built this sort of 3D model of Stanford campus in a single
picture. You can sort of walk into the ceiling, look” metadata={}

嘿，伙伴們，今天我們的AI探索之旅已經(jīng)圓滿(mǎn)結(jié)束。關(guān)于“面向開(kāi)發(fā)者的LLM入門(mén)教程-檢索(Retrieval)英文版”的內(nèi)容已經(jīng)分享給大家了。感謝你們的陪伴，希望這次旅程讓你對(duì)AI能夠更了解、更喜歡。謹(jǐn)記，精準(zhǔn)提問(wèn)是解鎖AI潛能的鑰匙哦！如果有小伙伴想要了解學(xué)習(xí)更多的AI知識(shí)，請(qǐng)關(guān)注我們的官網(wǎng)“AI智研社”，保證讓你收獲滿(mǎn)滿(mǎn)呦！