哈嘍！伙伴們，我是小智，你們的AI向?qū)?。歡迎來到每日的AI學(xué)習(xí)時間。今天，我們將一起深入AI的奇妙世界，探索“面向開發(fā)者的LLM入門教程-文檔加載英文版”，并學(xué)會本篇文章中所講的全部知識點。還是那句話“不必遠征未知，只需喚醒你的潛能！”跟著小智的步伐，我們終將學(xué)有所成，學(xué)以致用，并發(fā)現(xiàn)自身的更多可能性。話不多說，現(xiàn)在就讓我們開始這場激發(fā)潛能的AI學(xué)習(xí)之旅吧。

面向開發(fā)者的LLM入門教程-文檔加載英文版：

英文版

1.加載 PDF 文檔

from langchain.document_loaders import PyPDFLoader

# 創(chuàng)建一個 PyPDFLoader Class 實例，輸入為待加載的pdf文檔路徑
loader = PyPDFLoader(“docs/cs229_lectures/MachineLearning-Lecture01.pdf”)

# 調(diào)用 PyPDFLoader Class 的函數(shù) load對pdf文件進行加載
pages = loader.load()

2.探索加載的數(shù)據(jù)

print(“Type of pages: “, type(pages))
print(“Length of pages: “, len(pages))

page = pages[0]
print(“Type of page: “, type(page))
print(“Page_content: “, page.page_content[:500])
print(“Meta Data: “, page.metadata)

Type of pages:
Length of pages: 22
Type of page:
Page_content: MachineLearning-Lecture01
Instructor (Andrew Ng): Okay. Good morning. Welcome to CS229, the machine
learning class. So what I wanna do today is ju st spend a little time going over
the logistics
of the class, and then we’ll start to talk a bit about machine learning.
By way of introduction, my name’s Andrew Ng and I’ll be instru ctor for this
class. And so
I personally work in machine learning, and I’ ve worked on it for about 15 years
now, and
I actually think that machine learning i
Meta Data: {‘source’: ‘docs/cs229_lectures/MachineLearning-Lecture01.pdf’,
‘page’: 0}

3.加載 Youtube 音頻

# 注：由于該視頻較長，容易出現(xiàn)網(wǎng)絡(luò)問題，此處沒有運行，讀者可自行運行探索

from langchain.document_loaders.generic import GenericLoader
from langchain.document_loaders.parsers import OpenAIWhisperParser

from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader

url=”https://www.youtube.com/watch?v=jGwO_UgTS7I”
save_dir=”docs/youtube/”

# 創(chuàng)建一個 GenericLoader Class 實例
loader = GenericLoader(
#將鏈接url中的Youtube視頻的音頻下載下來,存在本地路徑save_dir
YoutubeAudioLoader([url],save_dir),

#使用OpenAIWhisperPaser解析器將音頻轉(zhuǎn)化為文本
OpenAIWhisperParser()
)

# 調(diào)用 GenericLoader Class 的函數(shù) load對視頻的音頻文件進行加載
docs = loader.load()

4.探索加載的數(shù)據(jù)

print(“Type of pages: “, type(pages))
print(“Length of pages: “, len(pages))

page = pages[0]
print(“Type of page: “, type(page))
print(“Page_content: “, page.page_content[:500])
print(“Meta Data: “, page.metadata)

5.加載網(wǎng)頁文檔

from langchain.document_loaders import WebBaseLoader

# 創(chuàng)建一個 WebBaseLoader Class 實例
url = “https://github.com/basecamp/handbook/blob/master/37signals-is-you.md”
header = {‘User-Agent’: ‘python-requests/2.27.1’,
‘Accept-Encoding’: ‘gzip, deflate, br’,
‘Accept’: ‘*/*’,
‘Connection’: ‘keep-alive’}
loader = WebBaseLoader(web_path=url,header_template=header)

# 調(diào)用 WebBaseLoader Class 的函數(shù) load對文件進行加載
pages = loader.load()

6.探索加載的數(shù)據(jù)

print(“Type of pages: “, type(pages))
print(“Length of pages: “, len(pages))
page = pages[0]
print(“Type of page: “, type(page))
print(“Page_content: “, page.page_content[:500])
print(“Meta Data: “, page.metadata)

Type of pages:
Length of pages: 1
Type of page:
Page_content: {“payload”:{“allShortcutsEnabled”:false,”fileTree”:{“”:{“items”:
[{“name”:”37signals-is-you.md”,”path”:”37signals-isyou.md”,”contentType”:”file”},
{“name”:”LICENSE.md”,”path”:”LICENSE.md”,”contentType”:”file”},
{“name”:”README.md”,”path”:”README.md”,”contentType”:”file”},{“name”:”benefitsand-perks.md”,”path”:”benefits-and-perks.md”,”contentType”:”file”},{“name”:”codeof-conduct.md”,”path”:”code-of-conduct.md”,”contentType”:”file”},
{“name”:”faq.md”,”path”:”faq.md”,”contentType”:”file”},{“name”:”ge
Meta Data: {‘source’:
‘https://github.com/basecamp/handbook/blob/master/37signals-is-you.md’}

進行進一步處理

import json
convert_to_json = json.loads(page.page_content)
extracted_markdow = convert_to_json[‘payload’][‘blob’][‘richText’]
print(extracted_markdow)

37signals Is You
Everyone working at 37signals represents 37signals. When a customer gets a
response from Merissa on support, Merissa is 37signals. When a customer reads a
tweet by Eron that our systems are down, Eron is 37signals. In those situations,
all the other stuff we do to cultivate our best image is secondary. What’s right
in front of someone in a time of need is what they’ll remember.
That’s what we mean when we say marketing is everyone’s responsibility, and that
it pays to spend the time to recognize that. This means avoiding the bullshit of
outage language and bending our policies, not just lending your ears. It means
taking the time to get the writing right and consider how you’d feel if you were
on the other side of the interaction.
The vast majority of our customers come from word of mouth and much of that word
comes from people in our audience. This is an audience we’ve been educating and
entertaining for 20 years and counting, and your voice is part of us now, whether
you like it or not! Tell us and our audience what you have to say!
This goes for tools and techniques as much as it goes for prose. 37signals not
only tries to out-teach the competition, but also out-share and out-collaborate.
We’re prolific open source contributors through Ruby on Rails, Trix, Turbolinks,
Stimulus, and many other projects. Extracting the common infrastructure that
others could use as well is satisfying, important work, and we should continue to
do that.
It’s also worth mentioning that joining 37signals can be all-consuming. We’ve
seen it happen. You dig 37signals, so you feel pressure to contribute, maybe
overwhelmingly so. The people who work here are some of the best and brightest in
our industry, so the self-imposed burden to be exceptional is real. But here’s
the thing: stop it. Settle in. We’re glad you love this job because we all do
too, but at the end of the day it’s a job. Do your best work, collaborate with
your team, write, read, learn, and then turn off your computer and play with your
dog. We’ll all be better for it.

7.加載 Notion 文檔

from langchain.document_loaders import NotionDirectoryLoader
loader = NotionDirectoryLoader(“docs/Notion_DB”)
pages = loader.load()

8.探索加載的數(shù)據(jù)

print(“Type of pages: “, type(pages))
print(“Length of pages: “, len(pages))

page = pages[0]
print(“Type of page: “, type(page))
print(“Page_content: “, page.page_content[:500])
print(“Meta Data: “, page.metadata)

Type of pages:
Length of pages: 51
Type of page:
Page_content: # #letstalkaboutstress
Let’s talk about stress. Too much stress.
We know this can be a topic.
So let’s get this conversation going.
[Intro: two things you should know]
(#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/Intro%20two%20things%20y
ou%20should%20know%20b5fd0c5393a9498b93396e79fe71e8bf.md)
[What is stress]
(#letstalkaboutstress%2064040a0733074994976118bbe0acc7fb/What%20is%20stress%20b19
8b685ed6a474ab14f6fafff7004b6.md)
[When is there too much stress?](#letstalkaboutstress%2
Meta Data: {‘source’: ‘docs/Notion_DB/#letstalkaboutstress
64040a0733074994976118bbe0acc7fb.md’}

嘿，伙伴們，今天我們的AI探索之旅已經(jīng)圓滿結(jié)束。關(guān)于“面向開發(fā)者的LLM入門教程-文檔加載英文版”的內(nèi)容已經(jīng)分享給大家了。感謝你們的陪伴，希望這次旅程讓你對AI能夠更了解、更喜歡。謹記，精準提問是解鎖AI潛能的鑰匙哦！如果有小伙伴想要了解學(xué)習(xí)更多的AI知識，請關(guān)注我們的官網(wǎng)“AI智研社”，保證讓你收獲滿滿呦！