哈嘍！伙伴們，我是小智，你們的AI向?qū)Аg迎來到每日的AI學(xué)習(xí)時間。今天，我們將一起深入AI的奇妙世界，探索“面向開發(fā)者的LLM入門課程-不存在簡單的正確答案英文版”，并學(xué)會本篇文章中所講的全部知識點。還是那句話“不必遠征未知，只需喚醒你的潛能！”跟著小智的步伐，我們終將學(xué)有所成，學(xué)以致用，并發(fā)現(xiàn)自身的更多可能性。話不多說，現(xiàn)在就讓我們開始這場激發(fā)潛能的AI學(xué)習(xí)之旅吧。

面向開發(fā)者的LLM入門課程-不存在簡單的正確答案英文版：

英文版

1.對問答系統(tǒng)提問

import utils_en
# 用戶消息
customer_msg = f”””
tell me about the smartx pro phone and the fotosnap camera, the dslr one.
Also, what TVs or TV related products do you have?”””
# 從問題中抽取商品名
products_by_category = utils_en.get_products_from_query(customer_msg)
# 將商品名轉(zhuǎn)化為列表
category_and_product_list = utils_en.read_string_to_list(products_by_category)
# 查找商品對應(yīng)的信息
product_info = utils_en.get_mentioned_product_info(category_and_product_list)
# 由信息生成回答
assistant_answer = utils_en.answer_user_msg(user_msg=customer_msg,
product_info=product_info)

print(assistant_answer)

Sure! Let me provide you with some information about the SmartX ProPhone and the
FotoSnap DSLR Camera.
The SmartX ProPhone is a powerful smartphone with advanced camera features. It
has a 6.1-inch display, 128GB storage, a 12MP dual camera, and supports 5G
connectivity. The SmartX ProPhone is priced at $899.99 and comes with a 1-year
warranty.
The FotoSnap DSLR Camera is a versatile camera that allows you to capture
stunning photos and videos. It features a 24.2MP sensor, 1080p video recording, a
3-inch LCD screen, and supports interchangeable lenses. The FotoSnap DSLR Camera
is priced at $599.99 and also comes with a 1-year warranty.
As for TVs and TV-related products, we have a range of options available. Some of
our popular TV models include the CineView 4K TV, CineView 8K TV, and CineView
OLED TV. We also have home theater systems like the SoundMax Home Theater and
SoundMax Soundbar. Could you please let me know your specific requirements or
preferences so that I can assist you better?

2.使用GPT評估

# 問題、上下文
cust_prod_info = {
‘customer_msg’: customer_msg,
‘context’: product_info
}

def eval_with_rubric(test_set, assistant_answer):
“””
使用 GPT API 評估生成的回答
參數(shù)：
test_set: 測試集
assistant_answer: 助手的回復(fù)
“””
cust_msg = test_set[‘customer_msg’]
context = test_set[‘context’]
completion = assistant_answer
# 要求 GPT 作為一個助手評估回答正確性
system_message = “””
You are an assistant that evaluates how well the customer service agent
answers a user question by looking at the context that the customer service
agent is using to generate its response.
“””
# 具體指令
user_message = f”””
You are evaluating a submitted answer to a question based on the context
that the agent uses to answer the question.
Here is the data:
[BEGIN DATA]
************
[Question]: {cust_msg}
************
[Context]: {context}
************
[Submission]: {completion}
************
[END DATA]
Compare the factual content of the submitted answer with the context.
Ignore any differences in style, grammar, or punctuation.
Answer the following questions:
– Is the Assistant response based only on the context provided? (Y or N)
– Does the answer include information that is not provided in the context? (Y
or N)
– Is there any disagreement between the response and the context? (Y or N)
– Count how many questions the user asked. (output a number)
– For each question that the user asked, is there a corresponding answer to
it?
Question 1: (Y or N)
Question 2: (Y or N)
…
Question N: (Y or N)
– Of the number of questions asked, how many of these questions were
addressed by the answer? (output a number)
“””
messages = [
{‘role’: ‘system’, ‘content’: system_message},
{‘role’: ‘user’, ‘content’: user_message}
]
response = get_completion_from_messages(messages)
return response

evaluation_output = eval_with_rubric(cust_prod_info, assistant_answer)
print(evaluation_output)

– Is the Assistant response based only on the context provided? (Y or N)
Y
– Does the answer include information that is not provided in the context? (Y or
N)
N
– Is there any disagreement between the response and the context? (Y or N)
N
– Count how many questions the user asked. (output a number)
2
– For each question that the user asked, is there a corresponding answer to it?
Question 1: Y
Question 2: Y
– Of the number of questions asked, how many of these questions were addressed by
the answer? (output a number)
2

3.評估生成回答與標(biāo)準(zhǔn)回答的差距

test_set_ideal = {
‘customer_msg’: “””
tell me about the smartx pro phone and the fotosnap camera, the dslr one.
Also, what TVs or TV related products do you have?”””,
‘ideal_answer’:”””
Of course! The SmartX ProPhone is a powerful
smartphone with advanced camera features.
For instance, it has a 12MP dual camera.
Other features include 5G wireless and 128GB storage.
It also has a 6.1-inch display. The price is $899.99.
The FotoSnap DSLR Camera is great for
capturing stunning photos and videos.
Some features include 1080p video,
3-inch LCD, a 24.2MP sensor,
and interchangeable lenses.
The price is 599.99.
For TVs and TV related products, we offer 3 TVs
All TVs offer HDR and Smart TV.
The CineView 4K TV has vibrant colors and smart features.
Some of these features include a 55-inch display,
‘4K resolution. It’s priced at 599.
The CineView 8K TV is a stunning 8K TV.
Some features include a 65-inch display and
8K resolution. It’s priced at 2999.99
The CineView OLED TV lets you experience vibrant colors.
Some features include a 55-inch display and 4K resolution.
It’s priced at 1499.99.
We also offer 2 home theater products, both which include bluetooth.
The SoundMax Home Theater is a powerful home theater system for
an immmersive audio experience.
Its features include 5.1 channel, 1000W output, and wireless subwoofer.
It’s priced at 399.99.
The SoundMax Soundbar is a sleek and powerful soundbar.
It’s features include 2.1 channel, 300W output, and wireless subwoofer.
It’s priced at 199.99
Are there any questions additional you may have about these products
that you mentioned here?
Or may do you have other questions I can help you with?
“””
}

def eval_vs_ideal(test_set, assistant_answer):
“””
評估回復(fù)是否與理想答案匹配
參數(shù)：
test_set: 測試集
assistant_answer: 助手的回復(fù)
“””
cust_msg = test_set[‘customer_msg’]
ideal = test_set[‘ideal_answer’]
completion = assistant_answer
system_message = “””
You are an assistant that evaluates how well the customer service agent
answers a user question by comparing the response to the ideal (expert)
response
Output a single letter and nothing else.
“””
user_message = f”””
You are comparing a submitted answer to an expert answer on a given question.
Here is the data:
[BEGIN DATA]
************
[Question]: {cust_msg}
************
[Expert]: {ideal}
************
[Submission]: {completion}
************
[END DATA]
Compare the factual content of the submitted answer with the expert answer.
Ignore any differences in style, grammar, or punctuation.
The submitted answer may either be a subset or superset of the expert answer,
or it may conflict with it. Determine which case applies.
Answer the question by selecting one of the following options:
(A) The submitted answer is a subset of the expert answer and is fully
consistent with it.
(B) The submitted answer is a superset of the expert answer and is fully
consistent with it.
(C) The submitted answer contains all the same details as the expert answer.
(D) There is a disagreement between the submitted answer and the expert
answer.
(E) The answers differ, but these differences don’t matter from the
perspective of factuality.
choice_strings: ABCDE
“””
messages = [
{‘role’: ‘system’, ‘content’: system_message},
{‘role’: ‘user’, ‘content’: user_message}
]
response = get_completion_from_messages(messages)
return response

print(assistant_answer)

Sure! Let me provide you with some information about the SmartX ProPhone and the
FotoSnap DSLR Camera.
The SmartX ProPhone is a powerful smartphone with advanced camera features. It
has a 6.1-inch display, 128GB storage, a 12MP dual camera, and supports 5G
connectivity. The SmartX ProPhone is priced at $899.99 and comes with a 1-year
warranty.
The FotoSnap DSLR Camera is a versatile camera that allows you to capture
stunning photos and videos. It features a 24.2MP sensor, 1080p video recording, a
3-inch LCD screen, and supports interchangeable lenses. The FotoSnap DSLR Camera
is priced at $599.99 and also comes with a 1-year warranty.
As for TVs and TV-related products, we have a range of options available. Some of
our popular TV models include the CineView 4K TV, CineView 8K TV, and CineView
OLED TV. We also have home theater systems like the SoundMax Home Theater and
SoundMax Soundbar. Could you please let me know your specific requirements or
preferences so that I can assist you better?

# 由于模型的更新，目前在原有 Prompt 上不再能夠正確判斷
eval_vs_ideal(test_set_ideal, assistant_answer)

‘D’

assistant_answer_2 = “life is like a box of chocolates”

eval_vs_ideal(test_set_ideal, assistant_answer_2)
# 對于明顯異常答案，GPT 判斷為不一致

‘D’

嘿，伙伴們，今天我們的AI探索之旅已經(jīng)圓滿結(jié)束。關(guān)于“面向開發(fā)者的LLM入門課程-不存在簡單的正確答案英文版”的內(nèi)容已經(jīng)分享給大家了。感謝你們的陪伴，希望這次旅程讓你對AI能夠更了解、更喜歡。謹(jǐn)記，精準(zhǔn)提問是解鎖AI潛能的鑰匙哦！如果有小伙伴想要了解學(xué)習(xí)更多的AI知識，請關(guān)注我們的官網(wǎng)“AI智研社”，保證讓你收獲滿滿呦！