第七色在线视频,2021少妇久久久久久久久久,亚洲欧洲精品成人久久av18,亚洲国产精品特色大片观看完整版,孙宇晨将参加特朗普的晚宴

111111111

1111111111111111

111111111111

爬蟲就是自動訪問互聯(lián)網(wǎng)? 并且提取數(shù)據(jù)的程序

1111111111

URL管理器

簡單爬蟲架構(gòu) - 運行流程

簡單的爬蟲架構(gòu)

做個截圖吧！

from baike_spider import html_downloader, html_outputer, url_manager
from lxml.html import html_parser
from astropy.units import count

class SpiderMain(object):
??? """爬蟲總調(diào)度程序，會使用HTML的下載器，解析器，輸出器來完成功能"""
??? def __init__(self):
??????? """在構(gòu)造函數(shù)中初始化所需要的各個對象"""
??????? self.urls = url_manager.UrlManager()????????????????? #URL管理器
??????? self.downloader = html_downloader.HtmlDownloader()??? #URL下載器
??????? self.parser = html_parser.HtmlParser()??????????????? #URL解析器
??????? self.output = html_outputer.HtmlOutputer()??????????? #URL輸出器
???
??? def craw(self,root_url):
??????? """爬蟲的調(diào)度程序，root_url就是入口url，將其添加到URL管理器"""
??????? count = 1
??????? self.urls.add_new_url(root_url)??????????????????????????? #添加一個新的未爬取的URL
??????? while self.urls.has_new_url():
??????????? try:
??????????????? """設(shè)置一個異常處理，用來處理網(wǎng)頁中已經(jīng)失效的URL和空的URL"""
??????????????? new_url = self.urls.get_new_url()????????????????? #獲取一個待爬取的URL
??????????????? print("crow %d: %s"%(count,new_url))
??????????????? html_cont = self.downloader.download(new_url)????? #下載URL頁面
??????????????? new_urls, new_data =self.parser.parse(new_url,html_cont)???
??????????????? #調(diào)用解析器解析頁面數(shù)據(jù)，得到新的URL列表和新的數(shù)據(jù)，傳入兩個參數(shù)，傳入該URL，和下載的數(shù)據(jù)
??????????????? self.urls.add_new_urls(new_urls)?????????????????? #新的URL列表補充到URLS
??????????????? self.output.collect_data(new_data)???????????????? #收集數(shù)據(jù)
???????????????
??????????????? if count == 100:
??????????????????? """設(shè)置一個循環(huán)控制URL的爬取個數(shù)"""
??????????????????? break
???????????????
??????????????? count? += 1
??????????? except:
??????????????? print("craw failed")
??????? self.outputer.output_html()

if __name__ == '__main__':
??? root_url = "??? obj_spider = SpiderMain()????? #調(diào)用主函數(shù)
??? obj_spider.craw(root_url)????? #啟動爬蟲

網(wǎng)頁解析器：從網(wǎng)頁中提取有價值數(shù)據(jù)的工具

python的網(wǎng)頁解析器：正則表達式; html.parser; Beautiful Soup; lxml