How to setup a script for you Apps/binary executables which are not installed via command line OR app store

I will use s3cmd as a example to show how to setup a script for your apps/binaries which are not installed via command line Or app store.

1) Download s3cmd package from http://s3tools.org/download

2) extract it, and you should get a folder called s3cmd-master

3) put the folder to anywhere you want (at here I move it to ~/)

4) create a script called s3cmd under dir /usr/local/bin/ and put the following lines as its content.

#!/bin/sh

~/s3cmd-master/s3cmd $*

5) make the created script executable: sudo chmod +x /usr/local/bin/s3cmd

DONE and you should able to use s3cmd command without specify the full path. 

Install Solr with Tomcat on CentOS

You must have at least version 1.6 in order to run Solr.

Installing Tomcat

yum -y install tomcat6 tomcat6-webapps tomcat6-admin-webapps
chkconfig tomcat6 on
service tomcat6 start

Use a web browser to check it is working correctly.

http://localhost:8080/

If you cannot connect, check your firewall rules

iptables -vnL --line-numbers

and make the necessary changes (Check these iptables snippets).

Config Tomcat

1. Tomcat user.

Ok, let’s go on. Add the following lines between the <tomcat-users> tag in /etc/tomcat6/tomcat-users.xml to add a manager role to your server:

<role rolename="manager" />
<role rolename="admin" />
<user username="tomcat" password="Pass.123" roles="manager,admin" />

Important: Make sure to change the previous username and password!

2. Change port to 8983.

Solr will be running on port 8983, so we need to change the default port of Tomcat to 8983 by modify /etc/tomcat6/server.xml and replace all 8080 port to 8983.

Restart Tomcat

service tomcat6 restart

3. Add a rule for port 8983 to iptables

sudo vi /etc/sysconfig/iptables

Add the following line into it.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8983 -j ACCEPT

Installing Apache Commons Loggins

Download Apache Commons Logging from http://commons.apache.org/proper/commons-logging/download_logging.cgi to your home directory.

Untar your file and copy the required files to your tomcat library folder:

cd
tar zxf commons-logging-1.1.3.tar.gz
cd commons-loggins-1.1.3
cp commons-logging-*.jar /usr/share/tomcat6/lib

Installing SLF4J

Download the right version from http://www.slf4j.org/ to your home directory.

Untar your file and copy the required files to your tomcat library folder:

cd
tar zxf slf4j-1.7.5.tar.gz
cd slf4j-1.7.5
cp slf4j-*.jar /usr/share/tomcat6/lib

Update: Latest versions include a couple Android files (slf4j-android-1.7.7.jar and slf4j-android-1.7.7-sources.jar). Do not copy them. They will be read by tomcat preventing Solr to start.

Installing Solr (Finally!)

Download Solr (version 4.5.1) from the Apache website to your home directory:

http://lucene.apache.org/solr/downloads.html

Uncompress it:

cd
tar zxf ~/solr-4.5.1.tar.gz

Copy (and rename) the included Solr war file (solr.war) to your Tomcat apps folder:

cp ~/solr-4.5.1/dist/solr-4.5.1.war /usr/share/tomcat6/webapps/solr.war

Now it’s time to create the folder that will keep your Solr index and documents. We also have to copy a basic Solr structure into it. Fortunately, Solr comes with a predefined structure and includes preconfigured files in it. You have to make sure that there is plenty of space wherever you place this folder, since it can grow a lot. By the way, you can place this folder out of your Tomcat webapps folder.

In my case, I have decided to put it in /home/solr. So, let’s do it:

mkdir /home/solr
cp -R ~/solr-4.5.1/example/solr/* /home/solr
chown -R tomcat /home/solr

Restart Tomcat.

service tomcat6 restart

Solr is not ready yet, however this last time we restarted Tomcat it unpackaged the solr.war file. We need to edit a file inside that unpackaged directory structure.

Edit your Solr web.xml file to let it know where your solar directory is located:

nano /usr/share/tomcat6/webapps/solr/WEB-INF/web.xml

Look for the following code and edit it (don’t forget to remove the comment markers!):

<env-entry>
    <env-entry-name>solr/home</env-entry-name>
    <env-entry-value>/home/solr</env-entry-value>
    <env-entry-type>java.lang.String</env-entry-type>
</env-entry>

Ok… so, restart Tomcat

service tomcat6 restart

On your web browser go to:

http://localhost:8983/solr

If everything is ok, your Solr instance should be ready to go.

Troubleshooting

I recommend you to completely stop and start Tomcat over and over and take a look at:

tail /var/log/tomcat6/catalina.out

If you do not restart Tomcat each time, errors will not be logged to the catalina.out file.

500 issue

Mostly you will get a error with solr and error message is like "SolrCore ‘collection1′ is not available due to init failure: Could not load config file /home/solr/collection1/solrconfig.xml"

This is because by default solr enable clustering and we can disable it by modifying /home/solr/collection1/conf/solrconfig.xml.

vi /home/solr/collection1/conf/solrconfig.xml

Search for <bool name="clustering">true</bool> and change it to false.

Restart tomcat.

Write permission issue to /home/solr folder

Modify the permission to /home/solr and make sure the user who running tomcat/solr has write permission to it.

References

http://andres.jaimes.net/878/setup-lucene-solr-centos-tomcat/

https://wiki.apache.org/solr/SolrLogging#Using_the_example_logging_setup_in_containers_other_than_Jetty

http://wiki.apache.org/solr/ClusteringComponent

Jenkins Plugins - Claim, Brakeman, Ruby Metrics

Claim

This is now installed on Jenkins & will let people claim broken builds - ie tell the world that you are going to fix it and how long you think it will take.

It needs to be enabled on a project by project basis, you do this when configuring your project by adding the 'Allow broken build claiming' post build action.

Brakeman

This is now installed on Jenkins & will display the output of the Brakeman static analysis gem.

To configure:

1 Add the brakeman gem to your Gemfile, eg

group :test do ... gem "brakeman" ... end

2 Add a line at the end of your Jenkins execute shell task to run brakeman, eg

bash -l -c 'brakeman -o brakeman-output.tabs'

3 Enable displaying the output by adding the 'Publish brakeman warnings' post build action to your project Jenkins configuration & setting the output file to point at the file you specified in step 2.

Ruby Metrics

The Jenkins Ruby Metrics plugin will display the output of rcov unit test coverage.

The easiest way we found to set this up:

1. Add simplecov & simplecov-rcov to your Gemfile, eg

group :test do ... gem "simplecov", ">=0.3.8", :require => false gem 'simplecov-rcov' ... end

2. Create a file called .simplecov in the root directory of your project

require 'simplecov-rcov' class SimpleCov::Formatter::MergedFormatter def format(result) SimpleCov::Formatter::HTMLFormatter.new.format(result) SimpleCov::Formatter::RcovFormatter.new.format(result) end end SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter SimpleCov.start 'rails' do # any custom configs like groups and filters can be here at a central place add_filter "/vendor/" end

3. Update spec/spec_helper & features/support/env.rb to require simplecov  ... add this after require 'rubygems' but before all else

require 'simplecov'

4. In Jenkins, enable displaying the output by adding the 'Publish rcov report' post build action in your project Jenkins configuration and pointing it at the coverage/rcov directory

5. See beautiful reports in Jenkins, and also locally when you've run the tests

6. Improve your tests!

.

Report on Vulnerable Gems with Bundle-Audit

Bundler Audit is a gem that reports on vulnerable gems in your Gemfile. Its similar to Gemnasium or https://hakiri.io/facets but can more easily be integrated into Jenkins. Its also recommended by the Brakeman people.

Install

Add the following to your Gemfile in the development group:

gem 'bundler-audit'

Run

bundle install

Run Locally

Run

bundle-audit update bundle-audit

This will output any vulnerable Gem versions you have, or a nice green message if you're ok

Integrate with Jenkins

To display the results on the project home page

Add the following to your "Execute Shell" build step:

bundle-audit update bundle-audit > bundle-audit.txt

Then under "Post build actions", add "Publish rich text message"

Select "confluence" markup, and paste the following:

h2. Bundle Audit Results ${FILE:bundle-audit.txt}

Now re-run your build and the results will display

Take it a step further and make Jenkins fail when there's vulnerable gems

Modify your execute shell build step to check the output of bundle-audit. Here's a simple example script which does this (see SnapDeploy for example)

#!/bin/bash cd ~ source /var/lib/jenkins/.rvm/scripts/rvm cd $WORKSPACE bundle install bundle update sqlite3 # Prepare db for testing RAILS_ENV=test bundle exec rake db:create db:migrate db:test:prepare --trace # Run rspec RAILS_ENV=test bundle exec rspec --no-color rspec_status=$? # Run cucumber RAILS_ENV=test bundle exec cucumber -p jenkins cucumber_status=$? # Run Brakeman brakeman -o brakeman-output.tabs --no-progress --separate-models # Run bundle audit bundle-audit update bundle-audit > bundle-audit.txt echo "BUNDLE AUDIT RESULTS:" cat bundle-audit.txt grep -Fxq "No unpatched versions found" bundle-audit.txt audit_status=$? # Fail if rspec or cucumber failed or there's vulnerable gems exit "$(($rspec_status + $cucumber_status + $audit_status))"

AVD CRASHES ON LAUNCH (EMULATOR64-ARM QUIT UNEXPECTEDLY)

IF you are doing Android development on a Mac using a second monitor connected, then you properly will get a error and the emulator is not started.  The error message “emulator64-arm quit unexpectedly”, this is a bug from Android dev tools and has beed filed https://code.google.com/p/android/issues/detail?id=38371.

The solution is:

cd ~/.android/avd/name_of_the_emulator.avd

vim emulator-user.ini

now change window.x = 0

DONE.

Host Maven repos on Github

Sometimes, we want to run a project as Maven repo to support other projects OR make it be available to other people to use as Maven dependencies. And I think hosting Maven artifacts on Github is more approachable than running a proper Maven Repository. In this post, I am going to show you how to do it step by step with a example.

1. Set up your Maven repo project on Github.

 You need to figure out where you are going to deploy and then host your project's artifacts. Here I have created a new repo on Github, and checked it out at the root of my dev directory.

dhcp123:server htxiong$ git clone git@github.com:htxiong/htxiong-mvn-repo.git
Initialized empty Git repository in ~/dev/htxiong-mvn-repo/.git/

warning: You appear to have cloned an empty repository.

2. Set up separate snapshots and releases directories.

This is not a technical necessity, but its good to maintain a ongoing project by separating snapshots and releases directories. 

dhcp123:server htxiong$ cd htxiong-mvn-repo
dhcp123:server htxiong$ mkdir snapshots
dhcp123:server htxiong$ mkdir releases

3. Deploy your project's artifacts to Maven repo.

In your project's pom.xml file, there is a configuration element <distributionManagement> that specifies the repositories to which one's project artifacts should be deployed. If you can not find it in your project's pom.xml, please add the following content into it and just before tag </project>.

<distributionManagement>

    <snapshotRepository>

        <id>mvn-repo-snapshot</id>

        <name>Maven repo example Snapshots</name>

         <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/snapshots</url>

    </snapshotRepository>

    <repository>

        <id>mvn-repo</id>

        <name>Maven repo example Releases</name>

        <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/releases</url>

    </repository>

</distributionManagement> 

Now let's build and deploy the project's artifacts by using altDeploymentRepository system property.

For snapshots:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=snapshot-repo::default::file:snapshots clean deploy

For releases:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=repo::default::file:releases clean deploy

And more information about DaltDeploymentRepository  can be found at http://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html#altDeploymentRepository.

4. Push to Github.

In the first 3 steps, we have built and deployed your project's artifacts in our local maven repo which is just like any other git repo, so changes need to be committed and pushed up in order to be useful.

dhcp123:server htxiong$ git add snapshots/*
dhcp123:server htxiong$ git add releases/*
dhcp123:server htxiong$ git commit -m ''add snapshots and releases"
dhcp123:server htxiong$ git push

5. Use your new Maven repo in other projects.

We have already deployed our project on Github as maven repo and the root of this repo will be at https://github.com/htxiong/htxiong-mvn-repo/raw/master/, then it is time now to use it in other projects by putting the following content into pom.xml.

For snapshots:

<repository>

    <id>-htxiong-mvn-repo</id>

    <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/snapshots</url>

</repository>

For releases:

<repository>

    <id>-htxiong-mvn-repo</id>

    <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/releases</url>

</repository>

DONE!