Nutch环境搭建
1. 环境准备HOST：Ubuntu12.04LTS
JDK： jdk-7u45-linux-i586.rpm
Nutch：apache-nutch-1.7-bin.tar.gz
Solr：solr-4.6.0-src.tgz
工作目录： /home/zephyr/
1.1JDK安装
为了下载快点，一下脑残下了rpm.
可Ubuntu上没有rpm工具apt-get install rpm 后rpm -ivh jdk-7u45-linux-i586.rpm
提示要使用alien
apt-get install alien alien -i jdk-7u45-linux-i586.rpm
alien -d jdk-7u45-linux-i586.rpm 生成deb文件 不可用
无奈，重新使用apt-get install oracle-java7-installer* 直接使用失败
add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java7-installer
到/etc/profile 文件的最后增加上以下几行
export JAVA_HOME=/usr/java/jdk1.7.0_45
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export NUTCH_HOME=/home/zephyr/apache-nutch-1.7
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$NUTCH_HOME/bin:$PATH
1.2Tomcat安装
apt-get install tomcat7 下载tomcat
修改tomcat的端口使用情况为
vi /etc/tomcat7/server.xml
<Connector port="8088" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
redirectPort="8443" />
/etc/init.d/tomcat7 start
1.3Nutch 安装
工作目录下解压 tar xzvf apache-nutch-1.7-bin.tar.gz
测试Nutch 是否能正常工作
在urls目录touch一个seed.txt echo http://nutch.apache.org/ >> seed.txt
修改conf/regex-urlfilter.txt
# accept anything else
+.
为
+^http://(*\.)*nutch.apache.org/
编辑conf/nutch-site.xml
<property>
<name>http.agent.name</name>
<value>My Nutch Spider</value>
</property>
.nutch crawl ../urls -dir ../test -depth 1 -topN 10
报错
java/lang/NoClassDefFoundError: java/lang/Object
Exception in thread "main" java.lang.InternalError: internal error: SHA-1 not available
修正JDK
cd /usr/java/jdk1.7.0_45/lib 将pack结尾的文件都转成 jar 如：unpack200 tools.pack tools.jar
/usr/java/jdk1.7.0_45/jre/lib 也做相同操作
再操作成功。
1.4 Solr安装
tar xzvf solr-4.6.0-src.tgz
ivy未安装，先安装ivy, apt-get install ivy
ant compile 失败，修正ln -s -T /usr/share/java/ivy.jar /usr/share/ant/lib
/ivy.jar 编译无响应
tar xzvf solr-4.6.0.tgz
把Solr 预设的core 范例collection1 复制为core-nutch
把Nutch 提供的conf/schema-solr4.xml 覆盖掉Solr core-nutch 的conf/schema.xml
将Solr core-nutch conf/schema.xml补上一行漏掉的栏位设定<field name=”_version_” type=”long” stored=”true” indexed=”true” multiValued=”false”/>
将core.properties中的name 改为 core-nutch
nutch crawl /home/zephyr/apache-nutch-1.7/urls -solr http://192.168.153.128:8983/solr/core-nutch/ -threads 20 -depth 1 -topN 10
也可以使用nutch solrindex http://192.168.153.128:8983/solr/core-nutch/ /home/zephyr/apache-nutch-1.7/test/crawldb -linkdb /home/zephyr/apache-nutch-1.7/test/linkdb/ /home/zephyr/apache-nutc
h-1.7/test/segments/*
1.5 solr通过tomcat7部署
apt-get install tomcat7-admin
/etc/tomcat7编辑tomcat-users.xml
<role rolename="manager-gui"/>
<role rolename="admin-gui"/>
<user username="tomcat" password="tomcat" roles="manager-gui,admin-gui"/>
cp /home/zephyr/solr-4.6.0/example/webapps/solr.war /var/lib/tomcat7/webapps/solr.war
cp –r /home/zephyr/solr-4.6.0/example/solr /var/lib/tomcat7/solr
cd /var/lib/tomcat7/conf/Catalina/localhost
vi solr.xml
<Context docBase="/var/lib/tomcat7/webapps/solr.war" debug="0" crossContext="true" >
<Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat7/solr" override="true" />
</Context>
cp /home/zephyr/solr-4.6.0/example/lib/ext/*.jar /usr/share/tomcat7/lib
cp /home/zephyr/solr-4.6.0/example/resources/log4j.properties /usr/share/tomcat7/lib/
vi log4j.properties 修改 solr.log=/var/logs/tomcat7/
否则solr无法在tomcat启动
“One of the things about running Solr in a container other than the embedded Jetty is that we need to setup the logging. This step is crucial, without doing this solr will never start and you will pull all of your hair out trying to figure out why”
启动运行提示出错：
SolrCore 'collection1' is not available due to init failure: Index locked for write for core collection1
cd /var/lib/tomcat7/solr/ collection1
chown tomcat7 data -R
cd /var/lib/tomcat7/solr/core-nutch
chown tomcat7 data –R
1.6 Nutch添加中文支持
编辑/home/zephyr/apache-nutch-1.7/conf/nutch-site.xml
<property>
<name>http.accept.language</name>
<value>ja-jp, en-us, zh-cn,en-gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept-Language” request header field.
This allows selecting non-English language as default one to retrieve.
It is a useful setting for search engines build for certain national group.
</description>
</property>
页:
[1]