scrapy框架中多个spider,tiems,pipelines的使用及运行方法

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

用scrapy只创建一个项目，创建多个spider，每个spider指定items,pipelines.启动爬虫时只写一个启动脚本就可以全部同时启动。

本文代码已上传至github,链接在文未。

一，创建多个spider的scrapy项目

scrapy startproject mymultispider

cd mymultispider

scrapy genspider myspd1 sina.com.cn

scrapy genspider myspd2 sina.com.cn

scrapy genspider myspd3 sina.com.cn

二，运行方法

1.为了方便观察，在spider中分别打印相关信息

import scrapy

class Myspd1Spider(scrapy.Spider):

    name = 'myspd1'

    allowed_domains = ['sina.com.cn']

    start_urls = ['http://sina.com.cn/']

    def parse(self, response):

        print('myspd1')

其他如myspd2,myspd3分别打印相关内容。

2.多个spider运行方法有两种，第一种写法比较简单，在项目目录下创建crawl.py文件，内容如下

from scrapy.crawler import CrawlerProcess

from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

# myspd1是爬虫名

process.crawl('myspd1')

process.crawl('myspd2')

process.crawl('myspd3')

process.start()

为了观察方便，可在settings.py文件中限定日志输出

LOG_LEVEL = 'ERROR'

右键运行此文件即可，输出如下

3.第二种运行方法为修改crawl源码，可以从官方的github中找到：https://github.com/scrapy/scrapy/blob/master/scrapy/commands/crawl.py

在spiders目录的同级目录下创建一个mycmd目录，并在该目录中创建一个mycrawl.py,将crawl源码复制进来，修改其中的run方法，改为如下内容

def run(self, args, opts):

    # 获取爬虫列表

    spd_loader_list = self.crawler_process.spider_loader.list()

    # 遍历各爬虫

    for spname in spd_loader_list or args:

        self.crawler_process.crawl(spname, **opts.spargs)

        print("此时启动的爬虫：" + spname)

    self.crawler_process.start()

在该文件的目录下创建初始化文件__init__.py

完成后机构目录如下

使用命令启动爬虫

scrapy mycrawl --nolog

输出如下：

三，指定items

1,这个比较简单，在items.py文件内创建相应的类，在spider中引入即可

items.py

import scrapy

class MymultispiderItem(scrapy.Item):

    # define the fields for your item here like:

    # name = scrapy.Field()

    pass

class Myspd1spiderItem(scrapy.Item):

    name = scrapy.Field()

class Myspd2spiderItem(scrapy.Item):

    name = scrapy.Field()

class Myspd3spiderItem(scrapy.Item):

    name = scrapy.Field()

spider内，例myspd1

# -*- coding: utf-8 -*-

import scrapy

from mymultispider.items import Myspd1spiderItem

class Myspd1Spider(scrapy.Spider):

    name = 'myspd1'

    allowed_domains = ['sina.com.cn']

    start_urls = ['http://sina.com.cn/']

    def parse(self, response):

        print('myspd1')

        item = Myspd1spiderItem()

        item['name'] = 'myspd1的pipelines'

        yield item

四，指定pipelines

1,这个也有两种方法，方法一，定义多个pipeline类：

pipelines.py文件内：

class Myspd1spiderPipeline(object):

    def process_item(self,item,spider):

        print(item['name'])

        return item

class Myspd2spiderPipeline(object):

    def process_item(self,item,spider):

        print(item['name'])

        return item

class Myspd3spiderPipeline(object):

    def process_item(self,item,spider):

        print(item['name'])

        return item

1.1settings.py文件开启管道

ITEM_PIPELINES = {

   # 'mymultispider.pipelines.MymultispiderPipeline': 300,

   'mymultispider.pipelines.Myspd1spiderPipeline': 300,

   'mymultispider.pipelines.Myspd2spiderPipeline': 300,

   'mymultispider.pipelines.Myspd3spiderPipeline': 300,

}

1.2spider中设置管道，例myspd1

# -*- coding: utf-8 -*-

import scrapy

from mymultispider.items import Myspd1spiderItem

class Myspd1Spider(scrapy.Spider):

    name = 'myspd1'

    allowed_domains = ['sina.com.cn']

    start_urls = ['http://sina.com.cn/']

    custom_settings = {

        'ITEM_PIPELINES': {'mymultispider.pipelines.Myspd1spiderPipeline': 300},

    }

    def parse(self, response):

        print('myspd1')

        item = Myspd1spiderItem()

        item['name'] = 'myspd1的pipelines'

        yield item

指定管道的代码

custom_settings = {

        'ITEM_PIPELINES': {'mymultispider.pipelines.Myspd1spiderPipeline': 300},

    }

1.3运行crawl文件，运行结果如下

2，方法二，在pipelines.py文件内判断是哪个爬虫的结果

2.1 pipelines.py文件内

class MymultispiderPipeline(object):

    def process_item(self, item, spider):

        if spider.name == 'myspd1':

            print('myspd1的pipelines')

        elif spider.name == 'myspd2':

            print('myspd2的pipelines')

        elif spider.name == 'myspd3':

            print('myspd3的pipelines')

        return item

2.2 settings.py文件内只开启MymultispiderPipeline这个管道文件

ITEM_PIPELINES = {

   'mymultispider.pipelines.MymultispiderPipeline': 300,

   # 'mymultispider.pipelines.Myspd1spiderPipeline': 300,

   # 'mymultispider.pipelines.Myspd2spiderPipeline': 300,

   # 'mymultispider.pipelines.Myspd3spiderPipeline': 300,

}

2.3spider中屏蔽掉指定pipelines的相关代码

# -*- coding: utf-8 -*-

import scrapy

from mymultispider.items import Myspd1spiderItem

class Myspd1Spider(scrapy.Spider):

    name = 'myspd1'

    allowed_domains = ['sina.com.cn']

    start_urls = ['http://sina.com.cn/']

    # custom_settings = {

    #     'ITEM_PIPELINES': {'mymultispider.pipelines.Myspd1spiderPipeline': 300},

    # }

    def parse(self, response):

        print('myspd1')

        item = Myspd1spiderItem()

        item['name'] = 'myspd1的pipelines'

        yield item

2.4 运行crawl.py文件，结果如下