python爬虫学习笔记（2）-----代理模式

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

一、UserAgent

UserAgent 中文意思是用户代理，简称UA，它是一个特殊字符串头，使得服务器能够识别用户

设置UA的两种方式：

1、heads

from urllib import request, error
if '__name__' == '__main__':
url = "http://www.baidu.com"
try:
headers = {}
headers['User-Agrnt'] = "User-Agent"," Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)"
req = request.Request(url, headers = headers)
rsp = request.urlopen(req)
html = rsp.read().decode()
print(html)
except error.HTTPError as e:
print(e)
except error.URLError as e:
print(e)
except Exception as e:
print(e)

2、使用add_header

from urllib import request, error
if __name__ = '__main__':
url = "http://www.baidu.com"
try:
req = request.Request(url)
req.add_header('User-Agent',"Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.33 Safari/534.3 SE 2.X MetaSr 1.0")
rsp = request.urlopen(req)
html = rsp.read().decode()
print(html)
except error.HTTPError as e:
print(e)
except error.URLEror as e:
print(e)
except Excecption as e:
print(e)

二、ProxyHandler（代理服务器）

由于很多网站会监测某一段时间内某个IP的访问次数，如果访问次数过多过快，它会禁止这个IP访问。所以我们可以设置一些代理服务器，就算IP被禁止依然可以换个IP继续访问。代理用来隐藏真实访问，代理也不许频繁访问某一个固定网址，所以，代理一定得多

基本使用步骤：

1、设置代理地址

2、创建ProxyHandler

3、创建Opener

4、安装Opener

获取代理服务器地址

- www.xicidaili.com

- www.goubanjia.com

from urllib import request, error
if __name__ == '__main__':
url = "http://www.baidu.com"
# 设置代理地址
proxy = {'http': '27.203.245.212:8060'}
# 创建ProxyHandler
proxy_handler = request.ProxyHandler(proxy)
# 创建Opener
opener = request.build_opener(proxy_handler)
# 安装Opener
request.install_opener(opener) try:
rsp = request.Request(url)
req = request.urlopen(rsp)
html = req.read().decode('utf-8')
print(html)
except error.HTTPError as e:
print(e)
except error.URLError as e:
print(e)
except Exception as e:
print(e)