scrapy相关：splash安装 A javascript rendering service 渲染

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

splash：美人鱼溅，泼

1.参考

Splash使用初体验

docker在windows下的安装

https://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/

Splash is our in-house solution for JavaScript rendering, implemented in Python using Twisted and QT. 官方博客介绍，splash 是 scrapinghub 的内部解决方案？？？

https://scrapinghub.com/

We're the creators and the main maintainers of Scrapy. 创始人和维护者...背后的大佬

github: scrapinghub/splash

Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python 3 using Twisted and QT5.

It's fast, lightweight and state-less which makes it easy to distribute. 用于渲染js页面

http://splash.readthedocs.io/en/latest/index.html

splash 官方文档

github: scrapy-plugins/scrapy-splash

This library provides Scrapy and JavaScript integration using Splash. 如何在 scrapy 中使用 splash

http://splash.readthedocs.io/en/stable/api.html#request-filters

Splash supports filtering requests based on Adblock Plus rules. 还没有搞定

2.安装使用

https://stackoverflow.com/questions/30345623/scraping-dynamic-content-using-python-scrapy

提到 ScrapyJS，但是链接地址跳转 https://github.com/scrapy-plugins/scrapy-splash#installation

https://pypi.python.org/pypi/scrapyjs

https://pypi.python.org/pypi/scrapy-splash

2.1 安装 scrapy-splash

C:\Users\win7>pip install scrapy-splash
Collecting scrapy-splash
Downloading scrapy_splash-0.7.2-py2.py3-none-any.whl
Installing collected packages: scrapy-splash
Successfully installed scrapy-splash-0.7.2

2.2 通过 docker 安装 image：scrapinghub/splash

官网找到下载链接

https://store.docker.com/editions/community/docker-ce-desktop-windows

Get Docker Community Edition for Windows

Docker for Windows is available for free.

Requires Microsoft Windows 10 Professional or Enterprise 64-bit. For previous versions get Docker Toolbox.

右键管理员安装，最好勾选非必要项？？？

右键管理员启动 Docker Quickstart Terminal ，提示没找到 bash.exe

输出：

Creating CA: C:\Users\win7\.docker\machine\certs\ca.pem
Creating client certificate: C:\Users\win7\.docker\machine\certs\cert.pem
Running pre-create checks...
(default) Image cache directory does not exist, creating it at C:\Users\win7\.docker\machine\cache...
(default) No default Boot2Docker ISO found locally, downloading the latest release...
(default) Latest release for github.com/boot2docker/boot2docker is v17.09.0-ce
(default) Downloading C:\Users\win7\.docker\machine\cache\boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v17.09.0-ce/boot2docker.iso...
(default) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Creating machine...
(default) Copying C:\Users\win7\.docker\machine\cache\boot2docker.iso to C:\Users\win7\.docker\machine\machines\default\boot2docker.iso...
(default) Creating VirtualBox VM...
(default) Creating SSH key...
(default) Starting the VM...
(default) Check network to re-create if needed...
(default) Windows might ask for the permission to create a network adapter. Sometimes, such confirmation window is minimized in the taskbar.
(default) Found a new host-only adapter: "VirtualBox Host-Only Ethernet Adapter #2"
(default) Windows might ask for the permission to configure a network adapter. Sometimes, such confirmation window is minimized in the taskbar.
(default) Windows might ask for the permission to configure a dhcp server. Sometimes, such confirmation window is minimized in the taskbar.
(default) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: D:\Program Files\Docker Toolbox\docker-machine.exe env default ## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.comStart interactive shellwin7@win7-PC MINGW64 ~
$ docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.4.89-boot2docker
Operating System: Boot2Docker 17.09.0-ce (TCL 7.2); HEAD : 06d5c35 - Wed Sep 27 23:22:43 UTC 2017
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.8MiB
Name: default
ID: O33J:6GDF:AQ6P:RBM7:6KLF:OZHY:2N3J:QZKV:YIJT:G3AI:XCPD:NZ3G
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 26
System Time: 2017-10-18T09:58:42.414047781Z
EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falsewin7@win7-PC MINGW64 ~
$ ipconfigWindows IP 配置以太网适配器 lan: 连接特定的 DNS 后缀 . . . . . . . :
本地链接 IPv6 地址. . . . . . . . : fe80::f950:bf55:726b:b7a6%14
IPv4 地址 . . . . . . . . . . . . : 192.168.144.100
子网掩码 . . . . . . . . . . . . : 255.255.255.0
默认网关. . . . . . . . . . . . . : 192.168.144.254以太网适配器 VirtualBox Host-Only Network #2: 连接特定的 DNS 后缀 . . . . . . . :
本地链接 IPv6 地址. . . . . . . . : fe80::1c18:13ad:7ed2:c0ff%29
IPv4 地址 . . . . . . . . . . . . : 192.168.99.1
子网掩码 . . . . . . . . . . . . : 255.255.255.0
默认网关. . . . . . . . . . . . . :隧道适配器 isatap.{CE007B04-2C7A-4A52-8BBF-1BCB4682EEB9}: 媒体状态 . . . . . . . . . . . . : 媒体已断开
连接特定的 DNS 后缀 . . . . . . . :隧道适配器 Teredo Tunneling Pseudo-Interface: 媒体状态 . . . . . . . . . . . . : 媒体已断开
连接特定的 DNS 后缀 . . . . . . . :隧道适配器 isatap.{93C68FD9-301C-484C-AFCB-5549CA24453B}: 媒体状态 . . . . . . . . . . . . : 媒体已断开
连接特定的 DNS 后缀 . . . . . . . :win7@win7-PC MINGW64 ~
$

里面重要信息:

(default) Copying C:\Users\win7\.docker\machine\cache\boot2docker.iso to C:\Users\win7\.docker\machine\machines\default\boot2docker.iso...
(default) Creating VirtualBox VM...docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com

putty 连接：

192.168.99.100
22docker
tcuser

第一次需要从docker hub下载相关镜像文件

sudo docker pull scrapinghub/splash

后面每次启动splash服务，并通过http，https，telnet提供服务

#通常一般使用http模式，可以只启动一个8050就好
#Splash 将运行在 0.0.0.0 at ports 8050 (http), 8051 (https) and 5023 (telnet).
sudo docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash

浏览器打开

http://192.168.99.100:8050