多个代理的管理 #128
Merged
多个代理的管理 #128
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Closed
code4craft added a commit that referenced this pull request May 27, 2014
code4craft added a commit that referenced this pull request May 27, 2014
| 很赞的功能，review了下，实现也不错。 但是我不知道怎么测试，看时间是2月份的，这段代码在自己的项目中已经运行一段时间了吧？ |
| 做了一点改动，将 if (site.getHttpProxyPool().isEnable()) { site.returnHttpProxyToPool((HttpHost) request.getExtra(Request.PROXY), (Integer) request .getExtra(Request.STATUS_CODE)); }这段代码从Spider挪到了HttpClientDownloader，因为代理本身是HttpClientDownloader的逻辑，不适合侵入到主流程内部。 另外statusCode和proxy这个可否考虑直接在HttpClientDownloader内部消化，不放到request内部？还是说，这个东东有特别的意义？ |
code4craft added a commit that referenced this pull request May 27, 2014
code4craft added a commit that referenced this pull request May 28, 2014
| 仔细看了一下，这里似乎PageProcessor也能对代理进行某种程度的操作，所以还必须得放到Spider中。先放回来，我想想有没有优化方案。 |
| 这个功能是我之前为了爬了一个网站开发的，后来一边改一边爬，最后效果挺好的，就发上来了。statusCode放在request内部是因为如果网站封了爬虫的时候，只有在解析网页的时候才能发现，比如要输验证码什么的，无法再HttpClientDownloader里面发现被封了，然后需要在pageprocessor 修改 page.statuscode传给request.stutuscode，这样返回代理池的时候才会对代理进行相应的修改。相关测试还没写 |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
先导入代理列表，各个线程均匀的使用这些代理，优先选择速度高的代理。