scrapy框架爬取蜂鸟网的人像图片

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

今天有点无聊,本来打算去蜂鸟网爬点图片存起来显得自己有点内涵,但是当我点开人像的时候就被里面的小姐姐所吸引了,下面就是整个爬图片的思路和过程了

第一步:先创建一个爬虫项目

scrapy startproject Feng

然后进入目录里面创建爬虫

好了爬虫项目创建完成了,接下来该去网页分析数据了

首先进入蜂鸟网找到人像摄影页面的网址添加到爬虫的start_urls里面

然后利用开发者工具分析网页页面的图片数据

发现图片的地址是在a标签里面的style属性里面的background-image:url,然后根据xpath找到该页面的所有人像图片的a标签里的style属性,获得的属性值有多余的background-image:url(),利用字符串的切片将这些去除掉,就可以获得完整的图片url

这样我就获取到了该页面所有的人像图片的url

然后在settings.py文件里面配置反爬机制

然后再引用scrapy.pipelines.images.ImagesPipeline

并给图片一个存储地址 IMAGE_STORE='./images' 即在当前目录下的images文件夹里面

然后再items.py文件里面设置对象

在parse里面导入items,然后创建item对象,然后将取到的图片地址赋给item对象在yield item

这样就可以下载当前页面的图片了

下面的翻页就是获取下一页的链接的xpath,用scrapy中的Request方法访问下一页,callback到parse就能循环翻页并下载

# -*- coding: utf-8 -*-

import scrapy

from ..items import FengniaoItem

class FengniaoSpider(scrapy.Spider):

    name = 'fengniao'

    allowed_domains = ['fengniao.com']

    start_urls = ['http://bbs.fengniao.com/forum/forum_101.html']

    def parse(self, response):

        image = response.xpath('//li/div/div/a/@style').extract()

        for i in image:

            item = FengniaoItem()

            img = i[21:-1]

            item['image_urls'] = [img]

            yield item

        nextpage = response.xpath('//a[text()="下一页"]/@href') .extract_first()

        yield scrapy.Request(url=nextpage,callback=self.parse,method='GET',dont_filter=True)

# -*- coding: utf-8 -*-

# Define here the models for your scraped items

#

# See documentation in:

# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy

class FengniaoItem(scrapy.Item):

    # define the fields for your item here like:

    # name = scrapy.Field()

    image_urls = scrapy.Field()

# -*- coding: utf-8 -*-

# Scrapy settings for Fengniao project

#

# For simplicity, this file contains only settings considered important or

# commonly used. You can find more settings consulting the documentation:

#

#     https://doc.scrapy.org/en/latest/topics/settings.html

#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html

#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'Fengniao'

SPIDER_MODULES = ['Fengniao.spiders']

NEWSPIDER_MODULE = 'Fengniao.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent

USER_AGENT = 'Fengniao (+http://www.yourdomain.com)'

# Obey robots.txt rules

ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)

CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)

# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay

# See also autothrottle settings and docs

DOWNLOAD_DELAY = 3

# The download delay setting will honor only one of:

#CONCURRENT_REQUESTS_PER_DOMAIN = 16

#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)

COOKIES_ENABLED = True

# Disable Telnet Console (enabled by default)

#TELNETCONSOLE_ENABLED = False

# Override the default request headers:

DEFAULT_REQUEST_HEADERS = {

  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',

  'Accept-Language': 'en',

}

# Enable or disable spider middlewares

# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html

#SPIDER_MIDDLEWARES = {

#    'Fengniao.middlewares.FengniaoSpiderMiddleware': 543,

#}

# Enable or disable downloader middlewares

# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html

#DOWNLOADER_MIDDLEWARES = {

#    'Fengniao.middlewares.FengniaoDownloaderMiddleware': 543,

#}

# Enable or disable extensions

# See https://doc.scrapy.org/en/latest/topics/extensions.html

#EXTENSIONS = {

#    'scrapy.extensions.telnet.TelnetConsole': None,

#}

# Configure item pipelines

# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html

ITEM_PIPELINES = {

   'Fengniao.pipelines.FengniaoPipeline': 300,

    'scrapy.pipelines.images.ImagesPipeline':1,

}

IMAGES_STORE = './images'

# Enable and configure the AutoThrottle extension (disabled by default)

# See https://doc.scrapy.org/en/latest/topics/autothrottle.html

#AUTOTHROTTLE_ENABLED = True

# The initial download delay

#AUTOTHROTTLE_START_DELAY = 5

# The maximum download delay to be set in case of high latencies

#AUTOTHROTTLE_MAX_DELAY = 60

# The average number of requests Scrapy should be sending in parallel to

# each remote server

#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

# Enable showing throttling stats for every response received:

#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)

# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings

#HTTPCACHE_ENABLED = True

#HTTPCACHE_EXPIRATION_SECS = 0

#HTTPCACHE_DIR = 'httpcache'

#HTTPCACHE_IGNORE_HTTP_CODES = []

#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

LOG_FILE = 'test.log'

FEED_EXPORT_ENCODING='utf-8'