aiohttp 支援非同步的網路請求模組

通常在進行網路資料採集時候我們會用到requests,urllib等模組,但是這些模組在使用中並不支援非同步,所以今天我們介紹一個支援非同步網路請求的模組aiohttp.

首先我們使用flask簡單的搭一個伺服器:

from flask import flask

def xiaohua():
return 'i am xiaohua'
def xiaohuang():
return 'i am xiaohuang'
def xiaoming():
return 'i am xiaoming'
if __name__ == '__main__':

import aiohttp

import time
import asyncio
async def get_page(url):               #加async關鍵字,使函式返回一個協程物件
async with aiohttp.clientsession() as session:  #建立clientsession物件
async with await session.get(url=url) as response:  #將url傳入get方法並賦值到response
page_text = await response.text()    #遇io手動掛起
print(page_text)
start = time.time()
urls = [
'','',
'ng',
'','',
'ng',
'','',
'ng',
]tasks = 
loop = asyncio.get_event_loop()    #建立事件迴圈
for url in urls:
c = get_page(url)     #接收協程物件
task = asyncio.ensure_future(c)   #對協程物件進一步封裝成future
loop.run_until_complete(asyncio.wait(tasks))   #將多個任務物件對應的列表註冊到事件迴圈中,需使用asyncio.wait進行逐一取值
print('總耗時：',time.time()-start)

i am xiaoming

i am xiaoming
i am xiaohuang
i am xiaoming
i am xiaohua
i am xiaohuang
i am xiaohuang
i am xiaohua
i am xiaohua
總耗時： 0.018949031829833984

從結果可以看出,通過使用aiohttp模組,訪問實現了非同步訪問,大大提高了效率.