正文
Linux wget 批量下载
提示：扫一扫查出行【扫一扫了解最新限行尾号】
复制提示
需求：已知50个pdf的URL地址，需要批量下载，该怎么办呢？
方案一：使用wget自带的一个功能 -i 选项 从指定文件中读取下载地址，这样的好处是一直是这一个wget进程下载所有pdf，不会来回的启、停止进程
[root@Jenkins tmp]# pwd
/root/tmp
[root@Jenkins tmp]# wc -l 50pdf.log
50pdf.log
[root@Jenkins tmp]# head - 50pdf.log
.pdf
.pdf
.pdf
[root@Jenkins tmp]# awk '{print "http://xxxxx/"$1}' 50pdf.log > download.log
[root@Jenkins tmp]# head - download.log
http://xxxxx/14788669468643331.pdf
http://xxxxx/1479035133045678.pdf
http://xxxxx/14799731544302441.pdf
[root@Jenkins tmp]# wget -i download.log
---- ::-- http://xxxxx/14788669468643331.pdf
Resolving nfs.htbaobao.com... 106.75.138.13
Connecting to nfs.htbaobao.com|106.75.138.13|:... connected.
HTTP request sent, awaiting response... OK
Length: (2.5M) [application/pdf]
Saving to: “.pdf”%[========================================================================================================================================================================>] ,, 244K/s in 10s -- :: ( KB/s) - “.pdf” saved [/]
.......................................中间省略
---- ::-- http://xxxxx/1481341338750833.pdf
Reusing existing connection to nfs.htbaobao.com:.
HTTP request sent, awaiting response... OK
Length: (149K) [application/pdf]
Saving to: “.pdf”%[========================================================================================================================================================================>] , 209K/s in .7s -- :: ( KB/s) - “.pdf” saved [/]FINISHED ---- ::--
Downloaded: files, 16M in 1m 13s ( KB/s)
[root@Jenkins tmp]# ls
14788669468643331.pdf 1481187682278708.pdf 1481262534034760.pdf 1481266593232456.pdf 1481340827926207.pdf 1481340948842260.pdf 1481341049634040.pdf 1481341172815801.pdf 1481341307823881.pdf
1479035133045678.pdf 1481193562811982.pdf 1481262611307371.pdf 1481267034803389.pdf 1481340853666343.pdf 1481340973957872.pdf 1481341112979143.pdf 1481341185245978.pdf 1481341338750833.pdf
14799731544302441.pdf 1481247789582233.pdf 1481262623674903.pdf 1481270022285676.pdf 1481340897933322.pdf 1481341008561312.pdf 1481341130545646.pdf 1481341216517700.pdf 50pdf.log
14799944743125144.pdf 1481262178457017.pdf 1481262846773279.pdf 1481286012498927.pdf 1481340922434822.pdf 1481341008584230.pdf 1481341134346522.pdf 1481341229730723.pdf download.log
1481034002739896.pdf 1481262229905206.pdf 1481265452669335.pdf 1481340787767089.pdf 1481340927135663.pdf 1481341022043499.pdf 1481341148759269.pdf 1481341244148718.pdf
1481095290513785.pdf 1481262241457479.pdf 1481265807661321.pdf 1481340826599027.pdf 1481340943094250.pdf 1481341045655154.pdf 1481341159027852.pdf 1481341261314587.pdf
在下载过程中打开另外一个窗口查看是否是同一个wget进程
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: wget -i download.log
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: wget -i download.log
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: wget -i download.log
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: wget -i download.log
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
[root@Jenkins ~]#
方案二：把这些URL地址放在一个文件里面，然后写个脚本直接for循环取一个URL地址交给wget下载，但是这样不好的是每下载一个pdf都会启动一个wget进程，下载完成后关闭wget进程 ......一直这样循环到最后一个，比较影响系统性能
[root@Jenkins tmp]# ls
50pdf.log download.log wget_pdf.sh
[root@Jenkins tmp]# cat wget_pdf.sh
#!/usr/bin/env bash
#
for url in `cat /root/tmp/download.log`;do
wget $url
done
[root@Jenkins tmp]# sh wget_pdf.sh
---- ::-- http://xxxxx/14788669468643331.pdf
Resolving nfs.htbaobao.com... 106.75.138.13
Connecting to nfs.htbaobao.com|106.75.138.13|:... connected.
HTTP request sent, awaiting response... OK
Length: (2.5M) [application/pdf]
Saving to: “.pdf”%[========================================================================================================================================================================>] ,, 230K/s in 11s -- :: ( KB/s) - “.pdf” saved [/]
......................................................中间省略
---- ::-- http://xxxxx/1481341338750833.pdf
Resolving nfs.htbaobao.com... 106.75.138.13
Connecting to nfs.htbaobao.com|106.75.138.13|:... connected.
HTTP request sent, awaiting response... OK
Length: (149K) [application/pdf]
Saving to: “.pdf”%[========================================================================================================================================================================>] , 184K/s in .8s -- :: ( KB/s) - “.pdf” saved [/][root@Jenkins tmp]# ls
.pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf
.pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf
.pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf 50pdf.log
.pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf download.log
.pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf wget_pdf.sh
.pdf .pdf .pdf .pdf .pdf .pdf .pdf .pdf
在下载过程中打开另外一个窗口查看是否是同一个wget进程
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/14788669468643331.pdf
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/1479035133045678.pdf
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/1479035133045678.pdf
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/14799731544302441.pdf
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/14799731544302441.pdf
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/14799944743125144.pdf
[root@Jenkins ~]# ps -ef|grep -v grep|grep wget
root : pts/ :: sh wget_pdf.sh
root : pts/ :: wget http://xxxxx/1481341307823881.pdf
小结：
1、使用方案一 只有一个进程下载，且在最后会显示总共下载了多少个文件，下载的总大小等信息
2、使用方案二 每次下载都会重新生成一个wget进程，上下文频繁切换
