核心代码java 核心代码怎么找

Java网络爬虫怎么实现？

网络爬虫是一个自动提取网页的程序，它为搜索引擎从万维网上下载网页，是搜索引擎的重要组成。\x0d\x0a传统爬虫从一个或若干初始网页的URL开始，获得初始网页上的URL，在抓取网页的过程中，不断从当前页面上抽取新的URL放入队列，直到满足系统的一定停止条件。对于垂直搜索来说，聚焦爬虫，即有针对性地爬取特定主题网页的爬虫，更为适合。\x0d\x0a\x0d\x0a以下是一个使用java实现的简单爬虫核心代码：\x0d\x0apublic void crawl() throws Throwable { \x0d\x0a while (continueCrawling()) { \x0d\x0a CrawlerUrl url = getNextUrl(); //获取待爬取队列中的下一个URL \x0d\x0a if (url != null) { \x0d\x0a printCrawlInfo(); \x0d\x0a String content = getContent(url); //获取URL的文本信息 \x0d\x0a \x0d\x0a //聚焦爬虫只爬取与主题内容相关的网页，这里采用正则匹配简单处理 \x0d\x0a if (isContentRelevant(content, this.regexpSearchPattern)) { \x0d\x0a saveContent(url, content); //保存网页至本地 \x0d\x0a \x0d\x0a //获取网页内容中的链接，并放入待爬取队列中 \x0d\x0a Collection urlStrings = extractUrls(content, url); \x0d\x0a addUrlsToUrlQueue(url, urlStrings); \x0d\x0a } else { \x0d\x0a System.out.println(url + " is not relevant ignoring ..."); \x0d\x0a } \x0d\x0a \x0d\x0a //延时防止被对方屏蔽 \x0d\x0a Thread.sleep(this.delayBetweenUrls); \x0d\x0a } \x0d\x0a } \x0d\x0a closeOutputStream(); \x0d\x0a}\x0d\x0aprivate CrawlerUrl getNextUrl() throws Throwable { \x0d\x0a CrawlerUrl nextUrl = null; \x0d\x0a while ((nextUrl == null) (!urlQueue.isEmpty())) { \x0d\x0a CrawlerUrl crawlerUrl = this.urlQueue.remove(); \x0d\x0a //doWeHavePermissionToVisit：是否有权限访问该URL，友好的爬虫会根据网站提供的"Robot.txt"中配置的规则进行爬取 \x0d\x0a //isUrlAlreadyVisited：URL是否访问过，大型的搜索引擎往往采用BloomFilter进行排重，这里简单使用HashMap \x0d\x0a //isDepthAcceptable：是否达到指定的深度上限。爬虫一般采取广度优先的方式。一些网站会构建爬虫陷阱（自动生成一些无效链接使爬虫陷入死循环），采用深度限制加以避免 \x0d\x0a if (doWeHavePermissionToVisit(crawlerUrl) \x0d\x0a (!isUrlAlreadyVisited(crawlerUrl)) \x0d\x0a isDepthAcceptable(crawlerUrl)) { \x0d\x0a nextUrl = crawlerUrl; \x0d\x0a // System.out.println("Next url to be visited is " + nextUrl); \x0d\x0a } \x0d\x0a } \x0d\x0a return nextUrl; \x0d\x0a}\x0d\x0aprivate String getContent(CrawlerUrl url) throws Throwable { \x0d\x0a //HttpClient4.1的调用与之前的方式不同 \x0d\x0a HttpClient client = new DefaultHttpClient(); \x0d\x0a HttpGet httpGet = new HttpGet(url.getUrlString()); \x0d\x0a StringBuffer strBuf = new StringBuffer(); \x0d\x0a HttpResponse response = client.execute(httpGet); \x0d\x0a if (HttpStatus.SC_OK == response.getStatusLine().getStatusCode()) { \x0d\x0a HttpEntity entity = response.getEntity(); \x0d\x0a if (entity != null) { \x0d\x0a BufferedReader reader = new BufferedReader( \x0d\x0a new InputStreamReader(entity.getContent(), "UTF-8")); \x0d\x0a String line = null; \x0d\x0a if (entity.getContentLength() 0) { \x0d\x0a strBuf = new StringBuffer((int) entity.getContentLength()); \x0d\x0a while ((line = reader.readLine()) != null) { \x0d\x0a strBuf.append(line); \x0d\x0a } \x0d\x0a } \x0d\x0a } \x0d\x0a if (entity != null) { \x0d\x0a nsumeContent(); \x0d\x0a } \x0d\x0a } \x0d\x0a //将url标记为已访问 \x0d\x0a markUrlAsVisited(url); \x0d\x0a return strBuf.toString(); \x0d\x0a}\x0d\x0apublic static boolean isContentRelevant(String content, \x0d\x0aPattern regexpPattern) { \x0d\x0a boolean retValue = false; \x0d\x0a if (content != null) { \x0d\x0a //是否符合正则表达式的条件 \x0d\x0a Matcher m = regexpPattern.matcher(content.toLowerCase()); \x0d\x0a retValue = m.find(); \x0d\x0a } \x0d\x0a return retValue; \x0d\x0a}\x0d\x0apublic List extractUrls(String text, CrawlerUrl crawlerUrl) { \x0d\x0a Map urlMap = new HashMap(); \x0d\x0a extractHttpUrls(urlMap, text); \x0d\x0a extractRelativeUrls(urlMap, text, crawlerUrl); \x0d\x0a return new ArrayList(urlMap.keySet()); \x0d\x0a} \x0d\x0aprivate void extractHttpUrls(Map urlMap, String text) { \x0d\x0a Matcher m = (text); \x0d\x0a while (m.find()) { \x0d\x0a String url = m.group(); \x0d\x0a String[] terms = url.split("a href=\""); \x0d\x0a for (String term : terms) { \x0d\x0a // System.out.println("Term = " + term); \x0d\x0a if (term.startsWith("http")) { \x0d\x0a int index = term.indexOf("\""); \x0d\x0a if (index 0) { \x0d\x0a term = term.substring(0, index); \x0d\x0a } \x0d\x0a urlMap.put(term, term); \x0d\x0a System.out.println("Hyperlink: " + term); \x0d\x0a } \x0d\x0a } \x0d\x0a } \x0d\x0a} \x0d\x0aprivate void extractRelativeUrls(Map urlMap, String text, \x0d\x0a CrawlerUrl crawlerUrl) { \x0d\x0a Matcher m = relativeRegexp.matcher(text); \x0d\x0a URL textURL = crawlerUrl.getURL(); \x0d\x0a String host = textURL.getHost(); \x0d\x0a while (m.find()) { \x0d\x0a String url = m.group(); \x0d\x0a String[] terms = url.split("a href=\""); \x0d\x0a for (String term : terms) { \x0d\x0a if (term.startsWith("/")) { \x0d\x0a int index = term.indexOf("\""); \x0d\x0a if (index 0) { \x0d\x0a term = term.substring(0, index); \x0d\x0a } \x0d\x0a String s = //" + host + term; \x0d\x0a urlMap.put(s, s); \x0d\x0a System.out.println("Relative url: " + s); \x0d\x0a } \x0d\x0a } \x0d\x0a } \x0d\x0a \x0d\x0a}\x0d\x0apublic static void main(String[] args) { \x0d\x0a try { \x0d\x0a String url = ""; \x0d\x0a Queue urlQueue = new LinkedList(); \x0d\x0a String regexp = "java"; \x0d\x0a urlQueue.add(new CrawlerUrl(url, 0)); \x0d\x0a NaiveCrawler crawler = new NaiveCrawler(urlQueue, 100, 5, 1000L, \x0d\x0a regexp); \x0d\x0a // boolean allowCrawl = crawler.areWeAllowedToVisit(url); \x0d\x0a // System.out.println("Allowed to crawl: " + url + " " + \x0d\x0a // allowCrawl); \x0d\x0a crawler.crawl(); \x0d\x0a } catch (Throwable t) { \x0d\x0a System.out.println(t.toString()); \x0d\x0a t.printStackTrace(); \x0d\x0a } \x0d\x0a}

相关问答

Q1: java设置servletoutputstream的文件名字

1. 核心代码

//设置Mime-Type

response.setContentType("application/msword");

//设置下载默认文件名

response.setHeader("Content-Disposition", "attachment;filename=123.doc");

登录后复制

完整代码

public RObject download(@PathVariable String id, HttpServletResponse response) {

File uploadFile = new File(id + ".doc");

//获取后缀名

String suffix = uploadFile.substring(uploadFile.lastIndexOf("."));

String responseType = ResponseTypeUtils.getResponseContentByType(suffix);

response.setContentType(responseType);

response.setHeader("Content-Disposition", "attachment;filename=" + System.currentTimeMillis() + suffix);

//输入流,通过输入流读取文件内容

try (FileInputStream fileInputStream = new FileInputStream(uploadFile)) {

ServletOutputStream outputStream = response.getOutputStream();

int len;

byte[] bytes = new byte[1024];

while ((len = fileInputStream.read(bytes)) != -1) {

outputStream.write(bytes, 0, len);

//刷新一下

outputStream.flush();

}

//关闭资源

outputStream.close();

return R.ok();

} catch (Exception e) {

throw new RuntimeException(e.getMessage(), e);

}

}

登录后复制

当然也可以通过NIO的方式实现代码，具体可参见HttpServletResponse 返回图片的几种方法对比，核心代码如下：

fis.getChannel().transferTo(0,fis.available(),Channels.newChannel(response.getOutputStream()));

登录后复制

3 参考状态码

文件扩展名 Content-Type(Mime-Type) 文件扩展名 Content-Type(Mime-Type)

.*（ 二进制流，不知道下载文件类型） application/octet-stream .tif image/tiff

.001 application/x-001 .301 application/x-301

.323 text/h323 .906 application/x-906

.907 drawing/907 .a11 application/x-a11

.acp audio/x-mei-aac .ai application/postscript

.aif audio/aiff .aifc audio/aiff

.aiff audio/aiff .anv application/x-anv

.asa text/asa .asf video/x-ms-asf

.asp text/asp .asx video/x-ms-asf

.au audio/basic .avi video/avi

.awf application/vnd.adobe.workflow .biz text/xml

.bmp application/x-bmp .bot application/x-bot

.c4t application/x-c4t .c90 application/x-c90

.cal application/x-cals .cat application/vnd.ms-pki.seccat

.cdf application/x-netcdf .cdr application/x-cdr

.cel application/x-cel .cer application/x-x509-ca-cert

.cg4 application/x-g4 .cgm application/x-cgm

.cit application/x-cit .class java/*

.cml text/xml .cmp application/x-cmp

.cmx application/x-cmx .cot application/x-cot

.crl application/pkix-crl .crt application/x-x509-ca-cert

.csi application/x-csi .css text/css

.cut application/x-cut .dbf application/x-dbf

.dbm application/x-dbm .dbx application/x-dbx

.dcd text/xml .dcx application/x-dcx

.der application/x-x509-ca-cert .dgn application/x-dgn

.dib application/x-dib .dll application/x-msdownload

.doc application/msword .dot application/msword

.docx application/msword .xlsx application/vnd.ms-excel

.drw application/x-drw .dtd text/xml

.dwf Model/vnd.dwf .dwf application/x-dwf

.dwg application/x-dwg .dxb application/x-dxb

.dxf application/x-dxf .edn application/vnd.adobe.edn

.emf application/x-emf .eml message/rfc822

.ent text/xml .epi application/x-epi

.eps application/x-ps .eps application/postscript

.etd application/x-ebx .exe application/x-msdownload

.fax image/fax .fdf application/vnd.fdf

.fif application/fractals .fo text/xml

.frm application/x-frm .g4 application/x-g4

.gbr application/x-gbr .pptx application/vnd.ms-powerpoint

.gif image/gif .gl2 application/x-gl2

.gp4 application/x-gp4 .hgl application/x-hgl

.hmr application/x-hmr .hpg application/x-hpgl

.hpl application/x-hpl .hqx application/mac-binhex40

.hrf application/x-hrf .hta application/hta

.htc text/x-component .htm text/html

.html text/html .htt text/webviewhtml

.htx text/html .icb application/x-icb

.ico image/x-icon .ico application/x-ico

.iff application/x-iff .ig4 application/x-g4

.igs application/x-igs .iii application/x-iphone

.img application/x-img .ins application/x-internet-signup

.isp application/x-internet-signup .IVF video/x-ivf

.java java/* .jfif image/jpeg

.jpe image/jpeg .jpe application/x-jpe

.jpeg image/jpeg .jpg image/jpeg

.jpg application/x-jpg .js application/x-javascript

.jsp text/html .la1 audio/x-liquid-file

.lar application/x-laplayer-reg .latex application/x-latex

.lavs audio/x-liquid-secure .lbm application/x-lbm

.lmsff audio/x-la-lms .ls application/x-javascript

.ltr application/x-ltr .m1v video/x-mpeg

.m2v video/x-mpeg .m3u audio/mpegurl

.m4e video/mpeg4 .mac application/x-mac

.man application/x-troff-man .math text/xml

.mdb application/msaccess .mdb application/x-mdb

.mfp application/x-shockwave-flash .mht message/rfc822

.mhtml message/rfc822 .mi application/x-mi

.mid audio/mid .midi audio/mid

.mil application/x-mil .mml text/xml

.mnd audio/x-musicnet-download .mns audio/x-musicnet-stream

.mocha application/x-javascript .movie video/x-sgi-movie

.mp1 audio/mp1 .mp2 audio/mp2

.mp2v video/mpeg .mp3 audio/mp3

.mp4 video/mpeg4 .mpa video/x-mpg

.mpd application/vnd.ms-project .mpe video/x-mpeg

.mpeg video/mpg .mpg video/mpg

.mpga audio/rn-mpeg .mpp application/vnd.ms-project

.mps video/x-mpeg .mpt application/vnd.ms-project

.mpv video/mpg .mpv2 video/mpeg

.mpw application/vnd.ms-project .mpx application/vnd.ms-project

.mtx text/xml .mxp application/x-mmxp

.net image/pnetvue .nrf application/x-nrf

.nws message/rfc822 .odc text/x-ms-odc

.out application/x-out .p10 application/pkcs10

.p12 application/x-pkcs12 .p7b application/x-pkcs7-certificates

.p7c application/pkcs7-mime .p7m application/pkcs7-mime

.p7r application/x-pkcs7-certreqresp .p7s application/pkcs7-signature

.pc5 application/x-pc5 .pci application/x-pci

.pcl application/x-pcl .pcx application/x-pcx

.pdf application/pdf .pdf application/pdf

.pdx application/vnd.adobe.pdx .pfx application/x-pkcs12

.pgl application/x-pgl .pic application/x-pic

.pko application/vnd.ms-pki.pko .pl application/x-perl

.plg text/html .pls audio/scpls

.plt application/x-plt .png image/png

.png application/x-png .pot application/vnd.ms-powerpoint

.ppa application/vnd.ms-powerpoint .ppm application/x-ppm

.pps application/vnd.ms-powerpoint .ppt application/vnd.ms-powerpoint

.ppt application/x-ppt .pr application/x-pr

.prf application/pics-rules .prn application/x-prn

.prt application/x-prt .ps application/x-ps

.ps application/postscript .ptn application/x-ptn

.pwz application/vnd.ms-powerpoint .r3t text/vnd.rn-realtext3d

.ra audio/vnd.rn-realaudio .ram audio/x-pn-realaudio

.ras application/x-ras .rat application/rat-file

.rdf text/xml .rec application/vnd.rn-recording

.red application/x-red .rgb application/x-rgb

.rjs application/vnd.rn-realsystem-rjs .rjt application/vnd.rn-realsystem-rjt

.rlc application/x-rlc .rle application/x-rle

.rm application/vnd.rn-realmedia .rmf application/vnd.adobe.rmf

.rmi audio/mid .rmj application/vnd.rn-realsystem-rmj

.rmm audio/x-pn-realaudio .rmp application/vnd.rn-rn_music_package

.rms application/vnd.rn-realmedia-secure .rmvb application/vnd.rn-realmedia-vbr

.rmx application/vnd.rn-realsystem-rmx .rnx application/vnd.rn-realplayer

.rp image/vnd.rn-realpix .rpm audio/x-pn-realaudio-plugin

.rsml application/vnd.rn-rsml .rt text/vnd.rn-realtext

.rtf application/msword .rtf application/x-rtf

.rv video/vnd.rn-realvideo .sam application/x-sam

.sat application/x-sat .sdp application/sdp

.sdw application/x-sdw .sit application/x-stuffit

.slb application/x-slb .sld application/x-sld

.slk drawing/x-slk .smi application/smil

.smil application/smil .smk application/x-smk

.snd audio/basic .sol text/plain

.sor text/plain .spc application/x-pkcs7-certificates

.spl application/futuresplash .spp text/xml

.ssm application/streamingmedia .sst application/vnd.ms-pki.certstore

.stl application/vnd.ms-pki.stl .stm text/html

.sty application/x-sty .svg text/xml

.swf application/x-shockwave-flash .tdf application/x-tdf

.tg4 application/x-tg4 .tga application/x-tga

.tif image/tiff .tif application/x-tif

.tiff image/tiff .tld text/xml

.top drawing/x-top .torrent application/x-bittorrent

.tsd text/xml .txt text/plain

.uin application/x-icq .uls text/iuls

.vcf text/x-vcard .vda application/x-vda

.vdx application/vnd.visio .vml text/xml

.vpg application/x-vpeg005 .vsd application/vnd.visio

.vsd application/x-vsd .vss application/vnd.visio

.vst application/vnd.visio .vst application/x-vst

.vsw application/vnd.visio .vsx application/vnd.visio

.vtx application/vnd.visio .vxml text/xml

.wav audio/wav .wax audio/x-ms-wax

.wb1 application/x-wb1 .wb2 application/x-wb2

.wb3 application/x-wb3 .wbmp image/vnd.wap.wbmp

.wiz application/msword .wk3 application/x-wk3

.wk4 application/x-wk4 .wkq application/x-wkq

.wks application/x-wks .wm video/x-ms-wm

.wma audio/x-ms-wma .wmd application/x-ms-wmd

.wmf application/x-wmf .wml text/vnd.wap.wml

.wmv video/x-ms-wmv .wmx video/x-ms-wmx

.wmz application/x-ms-wmz .wp6 application/x-wp6

.wpd application/x-wpd .wpg application/x-wpg

.wpl application/vnd.ms-wpl .wq1 application/x-wq1

.wr1 application/x-wr1 .wri application/x-wri

.wrk application/x-wrk .ws application/x-ws

.ws2 application/x-ws .wsc text/scriptlet

.wsdl text/xml .wvx video/x-ms-wvx

.xdp application/vnd.adobe.xdp .xdr text/xml

.xfd application/vnd.adobe.xfd .xfdf application/vnd.adobe.xfdf

.xhtml text/html .xls application/vnd.ms-excel

.xls application/x-xls .xlw application/x-xlw

.xml text/xml .xpl audio/scpls

.xq text/xml .xql text/xml

.xquery text/xml .xsd text/xml

.xsl text/xml .xslt text/xml

.xwd application/x-xwd .x_b application/x-x_b

.sis application/vnd.symbian.install .sisx application/vnd.symbian.install

.x_t application/x-x_t .ipa application/vnd.iphone

.apk application/vnd.android.package-archive .xap application

Q2: java语言中如何获取字符串中汉字的个数

【实例描述】

字符串中可以包含数字核心代码java，字母、汉字或者其他字符。使用Character类的isDigit()方法可以判断字符中的某个字符是否为数字核心代码java，使用Character类的isLetter()方法可以判断字符中的某个字符是否为字母。实例中将介绍一种方法用来判断字符串中的某个字符是否为汉字，通过此方法可以计算字符串中汉字的数量。实例的运行效果如图4.24所示。

【实现过程】

在Eclipse中新建项目ChineseCharacter，并在其中创建一个ChineseCharacter.java文件。在该类的主方法中创建标准输入流的扫描器对象，接收用户输入的字符串。我们在程序中使用matches()方法来统计该字符串中汉字的个数。核心代码如下所示：

protectedvoiddo_button_actionPerformed(ActionEvente){Stringtext=chineseArea.getText();//获取用户输入intamount=0;//创建汉字数量计数器for(inti=0;itext.length();i++){//遍历字符串每一个字符//使用正则表达式判断字符是否属于汉字编码booleanmatches=Pattern.matches("^[\u4E00-\u9FA5]{0,}$",""+text.charAt(i));if(matches){//如果是汉字amount++;//累加计数器}}umField.setText(amount+"");//在文本框显示汉字数量}

【代码解析】

本实例的关键点在于正则表达式的使用。Java提供了Pattern用于正则表达式的编译表示形式，该类提供的静态方法matches()可以执行正则表达式的匹配。该方法编译给定正则表达式并尝试给定输入与其匹配。如果要匹配的字符序列与正则表达式匹配则返回true，否则返回false。其声明语法如下：

publicstaticbooleanmatches(Stringregex,CharSequenceinput);【知识扩展】

使用正则表达式可以方便地进行字符串操作，正则表达式经常被用来验证用户输入的信息，如可以判断用户输入的格式是否正确。本实例中使用正则表达式来判断用户输入的字符串是否为汉字，如果为汉字则计数器加1，最后得到字符串中所有汉字的数量。

Q3: Java的核心代码是什么

《Java 核心代码》 是一本书，这就是核心代码java你问题核心代码java的答案。

这本书真的挺厚的，没办法给你打出来。

你要知道 Java 核心代码是什么，就把这本书从头到尾看一遍吧！

关于核心代码java和核心代码怎么找的介绍到此就结束了，不知道你从中找到你需要的信息了吗 ？如果你还想了解更多这方面的信息，记得收藏关注本站。