Scrapy框架CrawSpider原始碼分析

這一期的文章我們來分析一下 scrapy 框架中 crawspider 類的原始碼，希望能夠加深對其處理邏輯的理解。

crawlspider 繼承了 spider，其入口函式是 start_requests，該函式的 callback 為 crawlspider 下的 _parse 函式。

def

_parse
(self, response,
**kwargs)
:return self._parse_response(
response=response,
callback=self.parse_start_url,
cb_kwargs=kwargs,
follow=
true
,)

def

_parse_response
(self, response, callback, cb_kwargs, follow=
true):
if callback:
cb_res = callback(response,
**cb_kwargs)or(
) cb_res = self.process_results(response, cb_res)
for request_or_item in iterate_spider_output(cb_res)
:yield request_or_item
if follow and self._follow_links:
for request_or_item in self._requests_to_follow(response)
:yield request_or_item

def

parse_start_url
(self, response,
**kwargs)
:return
defprocess_results
(self, response, results)
:return results

rules =

( rule(linkextractor(allow=r'items/'
), callback=
'parse_iob'
, follow=
true),
)

def

_requests_to_follow
(self, response):if
notisinstance
(response, htmlresponse)
:return
seen =
set(
)for rule_index, rule in
enumerate
(self._rules)
: links =
[lnk for lnk in rule.link_extractor.extract_links(response)
if lnk not
in seen]
for link in rule.process_links(links)
: seen.add(link)
request = self._build_request(rule_index, link)
yield rule._process_request(request, response)

def

_callback
(self, response)
: rule = self._rules[response.meta[
'rule']]
return self._parse_response(response, rule.callback, rule.cb_kwargs, rule.follow)

linkextractor 引數用法

實際上是 lxmllinkextractor 類。

# top-level imports

from
.lxml import lxmllinkextractor as linkextractor

linkextractor 中引數分析：

注意：lxmllinkextractor 中 extract_links 函式：

def

extract_links
(self, response)
:"""returns a list of :class:`~scrapy.link.link` objects from the
specified :class:`response `.
only links that match the settings passed to the ``__init__`` method of
the link extractor are returned.
duplicate links are omitted.
"""base_url = get_base_url(response)
if self.restrict_xpaths:
docs =
[ subdoc
for x in self.restrict_xpaths
for subdoc in response.xpath(x)
]else
: docs =
[response.selector]
all_links =
for doc in docs:
links = self._extract_links(doc, response.url, response.encoding, base_url)
all_links.extend(self._process_links(links)
)return unique_list(all_links)

scrapy爬蟲框架（一） scrapy框架簡介

開啟命令列輸入如下命令 pip install scrapy安裝完成後，python會自動將 scrapy命令新增到環境變數中去，這時我們就可以使用 scrapy命令來建立我們的第乙個 scrapy專案了。開啟命令列，輸入如下命令 scrapy startproject yourproject這裡的...

scrapy爬蟲框架

作者經過幾周的python爬蟲實踐之後，深入學習了一下scrapy這個爬蟲框架，現將一些基本知識和 總結整理一下，以備後查。2.scrapy的命令列使用 這部分網上很多部落格都有總結，不需要背，理解會用主要的命令 startproject crawl fetch list genspider.即可，...

scrapy 爬蟲框架

1.安裝 公升級pip版本 pip install upgrade pip 通過pip安裝scrapy框架 pip install scrapy 安裝成功 只執行scrapy 進行測試是否安裝成功 2.scrapy startproject 爬蟲專案名稱 執行此命令,可以生成乙個爬蟲專案 會預先生成...