Wednesday, September 24, 2014

How to setup a script for you Apps/binary executables which are not installed via command line OR app store

I will use s3cmd as a example to show how to setup a script for your apps/binaries which are not installed via command line Or app store.

1) Download s3cmd package from http://s3tools.org/download
2) extract it, and you should get a folder called s3cmd-master
3) put the folder to anywhere you want (at here I move it to ~/)
4) create a script called s3cmd under dir /usr/local/bin/ and put the following lines as its content.
#!/bin/sh
~/s3cmd-master/s3cmd $*

5) make the created script executable: sudo chmod +x /usr/local/bin/s3cmd

DONE and you should able to use s3cmd command without specify the full path. 

Friday, June 20, 2014

Install Solr with Tomcat on CentOS

You must have at least version 1.6 in order to run Solr.

Installing Tomcat

yum -y install tomcat6 tomcat6-webapps tomcat6-admin-webapps
chkconfig tomcat6 on
service tomcat6 start
Use a web browser to check it is working correctly.
http://localhost:8080/
If you cannot connect, check your firewall rules
iptables -vnL --line-numbers
and make the necessary changes (Check these iptables snippets).

Config Tomcat

1. Tomcat user.

Ok, let’s go on. Add the following lines between the <tomcat-users> tag in /etc/tomcat6/tomcat-users.xml to add a manager role to your server:
<role rolename="manager" />
<role rolename="admin" />
<user username="tomcat" password="Pass.123" roles="manager,admin" />
Important: Make sure to change the previous username and password!

2. Change port to 8983.

Solr will be running on port 8983, so we need to change the default port of Tomcat to 8983 by modify /etc/tomcat6/server.xml and replace all 8080 port to 8983.
Restart Tomcat
service tomcat6 restart

3. Add a rule for port 8983 to iptables

sudo vi /etc/sysconfig/iptables
Add the following line into it.
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8983 -j ACCEPT

Installing Apache Commons Loggins

Download Apache Commons Logging from http://commons.apache.org/proper/commons-logging/download_logging.cgi to your home directory.
Untar your file and copy the required files to your tomcat library folder:
cd
tar zxf commons-logging-1.1.3.tar.gz
cd commons-loggins-1.1.3
cp commons-logging-*.jar /usr/share/tomcat6/lib

Installing SLF4J

Download the right version from http://www.slf4j.org/ to your home directory.
Untar your file and copy the required files to your tomcat library folder:
cd
tar zxf slf4j-1.7.5.tar.gz
cd slf4j-1.7.5
cp slf4j-*.jar /usr/share/tomcat6/lib
Update: Latest versions include a couple Android files (slf4j-android-1.7.7.jar and slf4j-android-1.7.7-sources.jar). Do not copy them. They will be read by tomcat preventing Solr to start.

Installing Solr (Finally!)

Download Solr (version 4.5.1) from the Apache website to your home directory:
http://lucene.apache.org/solr/downloads.html
Uncompress it:
cd
tar zxf ~/solr-4.5.1.tar.gz
Copy (and rename) the included Solr war file (solr.war) to your Tomcat apps folder:
cp ~/solr-4.5.1/dist/solr-4.5.1.war /usr/share/tomcat6/webapps/solr.war
Now it’s time to create the folder that will keep your Solr index and documents. We also have to copy a basic Solr structure into it. Fortunately, Solr comes with a predefined structure and includes preconfigured files in it. You have to make sure that there is plenty of space wherever you place this folder, since it can grow a lot. By the way, you can place this folder out of your Tomcat webapps folder.
In my case, I have decided to put it in /home/solr. So, let’s do it:
mkdir /home/solr
cp -R ~/solr-4.5.1/example/solr/* /home/solr
chown -R tomcat /home/solr
Restart Tomcat.
service tomcat6 restart
Solr is not ready yet, however this last time we restarted Tomcat it unpackaged the solr.war file. We need to edit a file inside that unpackaged directory structure.
Edit your Solr web.xml file to let it know where your solar directory is located:
nano /usr/share/tomcat6/webapps/solr/WEB-INF/web.xml
Look for the following code and edit it (don’t forget to remove the comment markers!):
<env-entry>
    <env-entry-name>solr/home</env-entry-name>
    <env-entry-value>/home/solr</env-entry-value>
    <env-entry-type>java.lang.String</env-entry-type>
</env-entry>
Ok… so, restart Tomcat
service tomcat6 restart
On your web browser go to:
http://localhost:8983/solr
If everything is ok, your Solr instance should be ready to go.
solr-admin

Troubleshooting

I recommend you to completely stop and start Tomcat over and over and take a look at:
tail /var/log/tomcat6/catalina.out
If you do not restart Tomcat each time, errors will not be logged to the catalina.out file.

500 issue

Mostly you will get a error with solr and error message is like "SolrCore ‘collection1′ is not available due to init failure: Could not load config file /home/solr/collection1/solrconfig.xml"
This is because by default solr enable clustering and we can disable it by modifying /home/solr/collection1/conf/solrconfig.xml.
vi /home/solr/collection1/conf/solrconfig.xml
Search for <bool name="clustering">true</bool> and change it to false.
Restart tomcat.

Write permission issue to /home/solr folder

Modify the permission to /home/solr and make sure the user who running tomcat/solr has write permission to it.

References

Thursday, June 19, 2014

Jenkins Plugins - Claim, Brakeman, Ruby Metrics

Claim

This is now installed on Jenkins & will let people claim broken builds - ie tell the world that you are going to fix it and how long you think it will take.
It needs to be enabled on a project by project basis, you do this when configuring your project by adding the 'Allow broken build claiming' post build action.

Brakeman

This is now installed on Jenkins & will display the output of the Brakeman static analysis gem.
To configure:
1 Add the brakeman gem to your Gemfile, eg
group :test do
  ...
  gem "brakeman"
  ...
end
2 Add a line at the end of your Jenkins execute shell task to run brakeman, eg
bash -l -c 'brakeman -o brakeman-output.tabs'
3 Enable displaying the output by adding the 'Publish brakeman warnings' post build action to your project Jenkins configuration & setting the output file to point at the file you specified in step 2.

Ruby Metrics

The Jenkins Ruby Metrics plugin will display the output of rcov unit test coverage.
The easiest way we found to set this up:
1. Add simplecov & simplecov-rcov to your Gemfile, eg
group :test do
  ...
  gem "simplecov", ">=0.3.8", :require => false
  gem 'simplecov-rcov'
  ...
end
2. Create a file called .simplecov in the root directory of your project
require 'simplecov-rcov'
class SimpleCov::Formatter::MergedFormatter
  def format(result)
    SimpleCov::Formatter::HTMLFormatter.new.format(result)
    SimpleCov::Formatter::RcovFormatter.new.format(result)
  end
end
SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter
SimpleCov.start 'rails' do
  # any custom configs like groups and filters can be here at a central place
  add_filter "/vendor/"
end
3. Update spec/spec_helper & features/support/env.rb to require simplecov  ... add this after require 'rubygems' but before all else
require 'simplecov'
4. In Jenkins, enable displaying the output by adding the 'Publish rcov report' post build action in your project Jenkins configuration and pointing it at the coverage/rcov directory
5. See beautiful reports in Jenkins, and also locally when you've run the tests
6. Improve your tests!
.

Report on Vulnerable Gems with Bundle-Audit

Bundler Audit is a gem that reports on vulnerable gems in your Gemfile. Its similar to Gemnasium or https://hakiri.io/facets but can more easily be integrated into Jenkins. Its also recommended by the Brakeman people.

Install

Add the following to your Gemfile in the development group:
  gem 'bundler-audit'
Run
bundle install

Run Locally

Run
bundle-audit update
bundle-audit
This will output any vulnerable Gem versions you have, or a nice green message if you're ok

Integrate with Jenkins

To display the results on the project home page

Add the following to your "Execute Shell" build step:
bundle-audit update
bundle-audit > bundle-audit.txt
Then under "Post build actions", add "Publish rich text message"
Select "confluence" markup, and paste the following:
h2. Bundle Audit Results
${FILE:bundle-audit.txt}
Now re-run your build and the results will display

Take it a step further and make Jenkins fail when there's vulnerable gems

Modify your execute shell build step to check the output of bundle-audit. Here's a simple example script which does this (see SnapDeploy for example)
#!/bin/bash
cd ~
source /var/lib/jenkins/.rvm/scripts/rvm
cd $WORKSPACE
bundle install
bundle update sqlite3


# Prepare db for testing
RAILS_ENV=test bundle exec rake db:create db:migrate db:test:prepare --trace


# Run rspec
RAILS_ENV=test bundle exec rspec --no-color
rspec_status=$?


# Run cucumber
RAILS_ENV=test bundle exec cucumber -p jenkins
cucumber_status=$?


# Run Brakeman
brakeman -o brakeman-output.tabs --no-progress --separate-models


# Run bundle audit
bundle-audit update
bundle-audit > bundle-audit.txt
echo "BUNDLE AUDIT RESULTS:"
cat bundle-audit.txt
grep -Fxq "No unpatched versions found" bundle-audit.txt
audit_status=$?


# Fail if rspec or cucumber failed or there's vulnerable gems
exit "$(($rspec_status + $cucumber_status + $audit_status))"

Thursday, February 13, 2014

AVD CRASHES ON LAUNCH (EMULATOR64-ARM QUIT UNEXPECTEDLY)

IF you are doing Android development on a Mac using a second monitor connected, then you properly will get a error and the emulator is not started.  The error message “emulator64-arm quit unexpectedly”, this is a bug from Android dev tools and has beed filed https://code.google.com/p/android/issues/detail?id=38371.

The solution is:

cd ~/.android/avd/name_of_the_emulator.avd
vim emulator-user.ini

now change window.x = 0


DONE.

Friday, January 3, 2014

Host Maven repos on Github

Sometimes, we want to run a project as Maven repo to support other projects OR make it be available to other people to use as Maven dependencies. And I think hosting Maven artifacts on Github is more approachable than running a proper Maven Repository. In this post, I am going to show you how to do it step by step with a example.

1. Set up your Maven repo project on Github.

 You need to figure out where you are going to deploy and then host your project's artifacts. Here I have created a new repo on Github, and checked it out at the root of my dev directory.

dhcp123:server htxiong$ git clone git@github.com:htxiong/htxiong-mvn-repo.git
Initialized empty Git repository in ~/dev/htxiong-mvn-repo/.git/

warning: You appear to have cloned an empty repository.

2. Set up separate snapshots and releases directories.

This is not a technical necessity, but its good to maintain a ongoing project by separating snapshots and releases directories. 

dhcp123:server htxiong$ cd htxiong-mvn-repo
dhcp123:server htxiong$ mkdir snapshots
dhcp123:server htxiong$ mkdir releases

3. Deploy your project's artifacts to Maven repo.

In your project's pom.xml file, there is a configuration element <distributionManagement> that specifies the repositories to which one's project artifacts should be deployed. If you can not find it in your project's pom.xml, please add the following content into it and just before tag </project>.


<distributionManagement>
    <snapshotRepository>
        <id>mvn-repo-snapshot</id>
        <name>Maven repo example Snapshots</name>
         <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/snapshots</url>
    </snapshotRepository>
    <repository>
        <id>mvn-repo</id>
        <name>Maven repo example Releases</name>
        <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/releases</url>
    </repository>
</distributionManagement> 

Now let's build and deploy the project's artifacts by using altDeploymentRepository system property.

For snapshots:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=snapshot-repo::default::file:snapshots clean deploy

For releases:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=repo::default::file:releases clean deploy

And more information about DaltDeploymentRepository  can be found at http://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html#altDeploymentRepository.

4. Push to Github.

In the first 3 steps, we have built and deployed your project's artifacts in our local maven repo which is just like any other git repo, so changes need to be committed and pushed up in order to be useful.

dhcp123:server htxiong$ git add snapshots/*
dhcp123:server htxiong$ git add releases/*
dhcp123:server htxiong$ git commit -m ''add snapshots and releases"
dhcp123:server htxiong$ git push

5. Use your new Maven repo in other projects.

We have already deployed our project on Github as maven repo and the root of this repo will be at https://github.com/htxiong/htxiong-mvn-repo/raw/master/, then it is time now to use it in other projects by putting the following content into pom.xml.

For snapshots:
<repository>
    <id>-htxiong-mvn-repo</id>
    <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/snapshots</url>

</repository>

For releases:
<repository>
    <id>-htxiong-mvn-repo</id>
    <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/releases</url>

</repository>



DONE!