利用xpath解析器爬取豆瓣電影top250

首先宣告需要用的庫，當然我還用到了os庫，將工作路徑修改到了我指定的路徑。os.chdir(r"c:\users…")

from lxml import etree

import requests 
import time
import json

def

get_page
(url)
: headers=
#請求頭可以通過f12快捷鍵獲得
response=requests.get(url,headers=headers)
return response.text

我們使用xpath來進行相應資訊的抽取。

def

get_message
(text)
""":parse text 是頁面返回的源**
使用xpath提取我們需要的電影名字，排名，影評等資訊"""
html=etree.xpath(text)
rank=html.xpath(
'//div[@class="pic"]/em/text()'
)name=html.xpath(
'//div[@class="hd"]/a/span[1]/text()'
)score=html.xpath(
'//div[@class="star"]/span[contains(@class,"rating_num")]/text()'
)words=html.xpath(
'//p[@class="quote"]/span/text()'
)for items in
range
(len
(rank)):
yield

def

write_file
(content)
:with
open
("result.txt"
,"a"
,encoding=
"gb18030"
)as f:
f.write(json.dumps(content,ensure_ascii=
false)+
'\n'
)

def

change_url
(start)
: url=
""+str
(start)
+"&filter="
return url
defmain
(start)
: url=change_url(start)
text=get_page(url)
for i in get_message(text)
:print
(i) write_file(i)
if __name__==
"__main__"
:for i in
range(10
):start=25*
10 main(start)
time.sleep(3)
#防止反爬蟲，如果速度過快可能會無響應，設定延時等待

from lxml import etree

import requests 
import time
import json
defget_page
(url)
: headers=
response=requests.get(url,headers=headers)
return response.text
defget_message
(text)：
html=etree.xpath(text)
rank=html.xpath(
'//div[@class="pic"]/em/text()'
) name=html.xpath(
'//div[@class="hd"]/a/span[1]/text()'
) score=html.xpath(
'//div[@class="star"]/span[contains(@class,"rating_num")]/text()'
) words=html.xpath(
'//p[@class="quote"]/span/text()'
)for items in
range
(len
(rank)):
yield
defwrite_file
(content)
:with
open
("result.txt"
,"a"
,encoding=
"gb18030"
)as f:
f.write(json.dumps(content,ensure_ascii=
false)+
'\n'
)def
change_url
(start)
: url=
""+str
(start)
+"&filter="
return url
defmain
(start)
: url=change_url(start)
text=get_page(url)
for i in get_message(text)
:print
(i) write_file(i)
if __name__==
"__main__"
:for i in
range(10
):start=25*
10 main(start)
time.sleep(3)
#防止反爬蟲，如果速度過快可能會無響應，設定延時等待

python爬蟲之scrapy爬取豆瓣電影（練習）

開發環境 windows pycharm mongodb scrapy 任務目標 任務目標 爬取 豆瓣電影top250 將資料儲存到mongodb中。items.py檔案 coding utf 8 define here the models for your scraped items see d...

使用xpath解析爬取鏈家

from urllib import request from time import sleep from lxml import etree import csv import redis import re 1 資料抓取 定義乙個函式,用於將頁碼,城市等資訊轉化為乙個request物件 def...

python利用Ajax分析方法爬取豆瓣劇情片排行

需要的庫 requests urllib.parse下的urlencode json csvtime 通過檢視network中的ajax請求的xhr檔案發現這條ajax請求包含的資訊如下 可以看到，包含了前20條電影的所有資訊 當再次向下滑動時，會出現新的xhr檔案 每個新的xhr檔案內包含了20條...