爬虫一、异步IO线程线程是计算机中工作的最小单元​ IO请求IO密集型时多线程更好计算密集型进程并发最好IO请求不涉及CPU自定义线程池进程进程默认有主线程可以有多线程共存并且共享内部资源自定义进程协程使用进程中一个线程去完成多个任务微线程伪线程GILpython特有用于在进程中对线程枷锁保证同一时刻只能有一个线程被CPU调度# Authorwylkjj# Date2020/2/24# -*- coding:utf-8 -*-importrequests# 创建多线程fromconcurrent.futuresimportThreadPoolExecutor# 创建多进程fromconcurrent.futuresimportProcessPoolExecutordefasync_url(url):try:responserequests.get(url)exceptExceptionase:print(异常结果,response.url,response.content)print(获取结果,response.url,response.content)url_list[http://www.baidu.com,http://www.chouti.com,http://www.bing.com,http://www.google.com,]# 线程池pool创建五个线程,IO请求线程更适合# GIL线程锁只针对cpu的调用权限针对IO请求不会锁住poolThreadPoolExecutor(5)# 进程池pools创建五个线程,进程浪费资源poolsProcessPoolExecutor(5)forurlinurl_list:print(开始请求,url)pool.submit(async_url,url)pool.shutdown(waitTrue)# 回调函数.add_done_callback(回调的函数)异步IO模块import asyncio缺点只提供TCP提供sleep不提供http​ 事件循环get_event_loop()​ asyncio.coroutine和yield from要同时配套使用固定写法​ 异步IOasynico aiohttpasynico requestgevent requestgevent request两个方法组合在一起后出现了一个grequeststwistedtornado异步非阻塞IO# Authorwylkjj# Date2020/2/24# -*- coding:utf-8 -*-# 异步IO模块importasyncioasyncio.coroutinedeffunc1():print(before...func1......)yieldfromasyncio.sleep(5)print(end...func1......)tasks[func1(),func1()]loopasyncio.get_event_loop()# 事件循环loop.run_until_complete(asyncio.gather(*tasks))# 把任务作为列表传进来loop.close()# Authorwylkjj# Date2020/2/25# -*- coding:utf-8 -*-importasyncioasyncio.coroutinedeffetch_async(host,url/):print(host,url)reader,writeryieldfromasyncio.open_connection(host,80)request_header_contentGET %s HTTP/1.0\r\nHost: %s\r\n\r\n%(url,host,)request_header_contentbytes(request_header_content,encodingutf-8)writer.write(request_header_content)yieldfromwriter.drain()textyieldfromreader.read()print(host,url,str(text,encodingutf-8))writer.close()tasks[fetch_async(www.cnblogs.com,/eric/),fetch_async(dig.chouti.com,/pic/show?nid4073644713430508lid10273091)]loopasyncio.get_event_loop()resultsloop.run_until_complete(asyncio.gather(*tasks))loop.close()# Authorwylkjj# Date2020/2/25# -*- coding:utf-8 -*-# 使用aiohttp和asyncio实现http请求 aiohttp亲importaiohttpimportasyncioasyncio.coroutinedeffetch_async(url):print(url)responseyieldfromaiohttp.request(GET,url)# data yield from response.read()# print(url, data)print(url,response)response.close()# Authorwylkjj# Date2020/2/25# -*- coding:utf-8 -*-# asyncio和requests配合使用也可以支持HTTP requests后importasyncioimportrequestsasyncio.coroutinedeffetch_async(func,*args):print(args)# 事件循环loopasyncio.get_event_loop()futureloop.run_in_executor(None,func,*args)responseyieldfromfutureprint(response.url,response.content)tasks[fetch_async(requests.get,http://www.cnblogs.com/eric/),fetch_async(requests.get,http://dig.chouti.com/pic/show?nid4073644713430508lid10273091)]loopasyncio.get_event_loop()resultsloop.run_until_complete(asyncio.gather(*tasks))loop.close()# Authorwylkjj# Date2020/2/25# -*- coding:utf-8 -*-importgeventfromgeventimportmonkey monkey.patch_all()importrequestsdeffetch_async(method,url,req_kwargs):print(method,url,req_kwargs)responserequests.request(methodmethod,urlurl,**req_kwargs)print(response.url,response.content)# ##### 发送请求 #####gevent.joinall([gevent.spawn(fetch_async,methodget,urlhttps://www.python.org/,req_kwargs{}),gevent.spawn(fetch_async,methodget,urlhttps://www.yahoo.com/,req_kwargs{}),gevent.spawn(fetch_async,methodget,urlhttps://github.com/,req_kwargs{}),])# pip3 install twisted# pip3 install wheel# b. 下载twisted http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted# c. 进入下载目录执行 pip3 install Twisted‑17.1.0‑cp35‑cp35m‑win_amd64.whlfromtwisted.web.clientimportgetPagefromtwisted.internetimportreactor REV_COUNTER0REQ_COUNTER0defcallback(contents):print(contents,)globalREV_COUNTER REV_COUNTER1ifREV_COUNTERREQ_COUNTER:reactor.stop()url_list[http://www.bing.com,http://www.baidu.com,]REQ_COUNTERlen(url_list)forurlinurl_list:print(url)deferredgetPage(bytes(url,encodingutf8))deferred.addCallback(callback)reactor.run()import socket它提供了标准的 BSD Sockets API可以访问底层操作系统Socket接口的全部方法。tronado框架原理自定义异步IO基于socketsetblockingFalseIO多路复用也是同步IOwhile True:r,w,e select.select([ ],[ ],[ ],1)关于IO的详情博客事件驱动IO模型https://www.cnblogs.com/wylshkjj/p/10896994.html二、scrapy框架scrapy框架的安装​ Linuxpip3 install scrapy​ Windows​ 1.​ pip3 install wheel​ 安装Twisted版本信息知识一个格式非正确版本​ a. http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted, 下载Twisted-19.1.0-cp37-cp37m-win_amd64.whl​ b. 进入文件所在目录​ c. pip3 install Twisted-19.1.0-cp37-cp37m-win_amd64.whl​ 2.pip3 install scrapy此版本与urllib3模块产生冲突如有此模块需要先卸载此模块​ 3.​ windows上scrapy依赖 https://sourceforge.net/projects/pywin32/files/项目的创建和执行scrapy使用方法创建新项目命令scrapy startproject scy 在想要创建的目录中执行此命令scy是项目名创建一个爬虫scrapy genspider example example.com 创建爬虫要先cd 到项目的目录中example是爬虫文件名字example.com 是所爬网页地址项目的执行命令scrapy crawl chouti (抽屉是所要执行的爬虫文件)过滤日志命令scrapy crawl chouti --nolog 过滤chouti 爬的数据日志查看爬虫模板命令scrapy genspider --list显示四个模板basiccrawlcsvfeedxmlfeed防止蜘蛛genspider 的权限robkts.txt属性在项目setting配置文件中修改ROBOTSTXT_OBEY属性使其值为ROBOTSTXT_OBEYFalseproject_name/scrapy.cfg 项目的主配置文件project_name/__init__.pyitems.py 设置数据存储模板用于结构化数据如Django的Modelpipelines.py 数据处理行为如一般结构化的数据持久化settings.py 真正配置文件如递归的层数并发数延迟下载等spiders/ 爬虫目录如创建文件编写爬虫规则__init__.py爬虫1.py爬虫2.py注意创建爬虫还是要在命令行创建运行项目运行爬虫文件都要在命令行执行# 部分项目代码展示爬取优美图库图片# -*- coding: utf-8 -*-importscrapyfromscrapy.httpimportRequestfrombs4importBeautifulSoupclassUmeiSpider(scrapy.Spider):nameumeiallowed_domains[umei.cc]start_urls[https://www.umei.cc/meinvtupian/meinvxiezhen/1.htm]visited_setset()defparse(self,response):self.visited_set.add(response.url)# 已经爬取的网页# 1.将当前页所有的meizi图片爬下来# 获取a标签并且属性为 class TypeBigPicsmain_pageBeautifulSoup(response.text,html.parser)item_listmain_page.find_all(a,attrs{class:TypeBigPics})foriteminitem_list:itemitem.find_all(img,)print(item)# 2.获取https://www.umei.cc/meinvtupian/meinvxiezhen/\d.htmpage_listmain_page.find_all(div,attrs{class:NewPages})a_urlshttps://www.umei.cc/meinvtupian/meinvxiezhen/a_listpage_list[0].find_all(a)a_hrefset()foraina_list:aa.get(href)ifa:a_href.add(a_urlsa)else:passforiina_href:ifiinself.visited_set:passelse:objRequest(urli,methodGET,callbackself.parse)yieldobjprint(obj:,obj)