Friday, June 20, 2014

Install Solr with Tomcat on CentOS

You must have at least version 1.6 in order to run Solr.

Installing Tomcat

yum -y install tomcat6 tomcat6-webapps tomcat6-admin-webapps
chkconfig tomcat6 on
service tomcat6 start

Use a web browser to check it is working correctly.

http://localhost:8080/

If you cannot connect, check your firewall rules

iptables -vnL --line-numbers

and make the necessary changes (Check these iptables snippets).

Config Tomcat

1. Tomcat user.

Ok, let’s go on. Add the following lines between the <tomcat-users> tag in /etc/tomcat6/tomcat-users.xml to add a manager role to your server:

<role rolename="manager" />
<role rolename="admin" />
<user username="tomcat" password="Pass.123" roles="manager,admin" />

Important: Make sure to change the previous username and password!

2. Change port to 8983.

Solr will be running on port 8983, so we need to change the default port of Tomcat to 8983 by modify /etc/tomcat6/server.xml and replace all 8080 port to 8983.

Restart Tomcat

service tomcat6 restart

3. Add a rule for port 8983 to iptables

sudo vi /etc/sysconfig/iptables

Add the following line into it.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8983 -j ACCEPT

Installing Apache Commons Loggins

Download Apache Commons Logging from http://commons.apache.org/proper/commons-logging/download_logging.cgi to your home directory.

Untar your file and copy the required files to your tomcat library folder:

cd
tar zxf commons-logging-1.1.3.tar.gz
cd commons-loggins-1.1.3
cp commons-logging-*.jar /usr/share/tomcat6/lib

Installing SLF4J

Download the right version from http://www.slf4j.org/ to your home directory.

Untar your file and copy the required files to your tomcat library folder:

cd
tar zxf slf4j-1.7.5.tar.gz
cd slf4j-1.7.5
cp slf4j-*.jar /usr/share/tomcat6/lib

Update: Latest versions include a couple Android files (slf4j-android-1.7.7.jar and slf4j-android-1.7.7-sources.jar). Do not copy them. They will be read by tomcat preventing Solr to start.

Installing Solr (Finally!)

Download Solr (version 4.5.1) from the Apache website to your home directory:

http://lucene.apache.org/solr/downloads.html

Uncompress it:

cd
tar zxf ~/solr-4.5.1.tar.gz

Copy (and rename) the included Solr war file (solr.war) to your Tomcat apps folder:

cp ~/solr-4.5.1/dist/solr-4.5.1.war /usr/share/tomcat6/webapps/solr.war

Now it’s time to create the folder that will keep your Solr index and documents. We also have to copy a basic Solr structure into it. Fortunately, Solr comes with a predefined structure and includes preconfigured files in it. You have to make sure that there is plenty of space wherever you place this folder, since it can grow a lot. By the way, you can place this folder out of your Tomcat webapps folder.

In my case, I have decided to put it in /home/solr. So, let’s do it:

mkdir /home/solr
cp -R ~/solr-4.5.1/example/solr/* /home/solr
chown -R tomcat /home/solr

Restart Tomcat.

service tomcat6 restart

Solr is not ready yet, however this last time we restarted Tomcat it unpackaged the solr.war file. We need to edit a file inside that unpackaged directory structure.

Edit your Solr web.xml file to let it know where your solar directory is located:

nano /usr/share/tomcat6/webapps/solr/WEB-INF/web.xml

Look for the following code and edit it (don’t forget to remove the comment markers!):

<env-entry>
    <env-entry-name>solr/home</env-entry-name>
    <env-entry-value>/home/solr</env-entry-value>
    <env-entry-type>java.lang.String</env-entry-type>
</env-entry>

Ok… so, restart Tomcat

service tomcat6 restart

On your web browser go to:

http://localhost:8983/solr

If everything is ok, your Solr instance should be ready to go.

Troubleshooting

I recommend you to completely stop and start Tomcat over and over and take a look at:

tail /var/log/tomcat6/catalina.out

If you do not restart Tomcat each time, errors will not be logged to the catalina.out file.

500 issue

Mostly you will get a error with solr and error message is like "SolrCore ‘collection1′ is not available due to init failure: Could not load config file /home/solr/collection1/solrconfig.xml"

This is because by default solr enable clustering and we can disable it by modifying /home/solr/collection1/conf/solrconfig.xml.

vi /home/solr/collection1/conf/solrconfig.xml

Search for <bool name="clustering">true</bool> and change it to false.

Restart tomcat.

Write permission issue to /home/solr folder

Modify the permission to /home/solr and make sure the user who running tomcat/solr has write permission to it.

References

http://andres.jaimes.net/878/setup-lucene-solr-centos-tomcat/

https://wiki.apache.org/solr/SolrLogging#Using_the_example_logging_setup_in_containers_other_than_Jetty

http://wiki.apache.org/solr/ClusteringComponent

Thursday, June 19, 2014

Jenkins Plugins - Claim, Brakeman, Ruby Metrics

Claim

This is now installed on Jenkins & will let people claim broken builds - ie tell the world that you are going to fix it and how long you think it will take.

It needs to be enabled on a project by project basis, you do this when configuring your project by adding the 'Allow broken build claiming' post build action.

Brakeman

This is now installed on Jenkins & will display the output of the Brakeman static analysis gem.

To configure:

1 Add the brakeman gem to your Gemfile, eg

group :test do
  ...
  gem "brakeman"
  ...
end

2 Add a line at the end of your Jenkins execute shell task to run brakeman, eg

bash -l -c 'brakeman -o brakeman-output.tabs'

3 Enable displaying the output by adding the 'Publish brakeman warnings' post build action to your project Jenkins configuration & setting the output file to point at the file you specified in step 2.

Ruby Metrics

The Jenkins Ruby Metrics plugin will display the output of rcov unit test coverage.

The easiest way we found to set this up:

1. Add simplecov & simplecov-rcov to your Gemfile, eg

group :test do
  ...
  gem "simplecov", ">=0.3.8", :require => false
  gem 'simplecov-rcov'
  ...
end

2. Create a file called .simplecov in the root directory of your project

require 'simplecov-rcov'
class SimpleCov::Formatter::MergedFormatter
  def format(result)
    SimpleCov::Formatter::HTMLFormatter.new.format(result)
    SimpleCov::Formatter::RcovFormatter.new.format(result)
  end
end
SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter
SimpleCov.start 'rails' do
  # any custom configs like groups and filters can be here at a central place
  add_filter "/vendor/"
end

3. Update spec/spec_helper & features/support/env.rb to require simplecov ... add this after require 'rubygems' but before all else

require 'simplecov'

4. In Jenkins, enable displaying the output by adding the 'Publish rcov report' post build action in your project Jenkins configuration and pointing it at the coverage/rcov directory

5. See beautiful reports in Jenkins, and also locally when you've run the tests

6. Improve your tests!

Report on Vulnerable Gems with Bundle-Audit

Bundler Audit is a gem that reports on vulnerable gems in your Gemfile. Its similar to Gemnasium or https://hakiri.io/facets but can more easily be integrated into Jenkins. Its also recommended by the Brakeman people.

Install

Add the following to your Gemfile in the development group:

  gem 'bundler-audit'

Run

bundle install

Run Locally

Run

bundle-audit update
bundle-audit

This will output any vulnerable Gem versions you have, or a nice green message if you're ok

Integrate with Jenkins

To display the results on the project home page

Add the following to your "Execute Shell" build step:

bundle-audit update
bundle-audit > bundle-audit.txt

Then under "Post build actions", add "Publish rich text message"

Select "confluence" markup, and paste the following:

h2. Bundle Audit Results
${FILE:bundle-audit.txt}

Now re-run your build and the results will display

Take it a step further and make Jenkins fail when there's vulnerable gems

Modify your execute shell build step to check the output of bundle-audit. Here's a simple example script which does this (see SnapDeploy for example)

#!/bin/bash
cd ~
source /var/lib/jenkins/.rvm/scripts/rvm
cd $WORKSPACE
bundle install
bundle update sqlite3


# Prepare db for testing
RAILS_ENV=test bundle exec rake db:create db:migrate db:test:prepare --trace


# Run rspec
RAILS_ENV=test bundle exec rspec --no-color
rspec_status=$?


# Run cucumber
RAILS_ENV=test bundle exec cucumber -p jenkins
cucumber_status=$?


# Run Brakeman
brakeman -o brakeman-output.tabs --no-progress --separate-models


# Run bundle audit
bundle-audit update
bundle-audit > bundle-audit.txt
echo "BUNDLE AUDIT RESULTS:"
cat bundle-audit.txt
grep -Fxq "No unpatched versions found" bundle-audit.txt
audit_status=$?


# Fail if rspec or cucumber failed or there's vulnerable gems
exit "$(($rspec_status + $cucumber_status + $audit_status))"

Sean Xiong