Friday, February 20, 2015

Use both optional arguments and positional arguments in Python apps

Normally python scripts accept arguments from command line, and in Python you can achieve that by using ArgumentParser.

e.g. Your Python script named MyApp and it accepts 4 arguments (foo1, foo2, foo3 and foo4) from command line, then your code will be something like:


And all these 4 arguments are optional, if in the case of foo1 is required, you can set required=True.


This works, but its not a good solution (more details can be found

So, set foo1 as positional argument might be better, and you can do like:


Then your python script can be ran: python -m MyApp value1 --foo2=value2 --foo3=value3 --foo4=value4.

Wednesday, September 24, 2014

How to setup a script for you Apps/binary executables which are not installed via command line OR app store

I will use s3cmd as a example to show how to setup a script for your apps/binaries which are not installed via command line Or app store.

1) Download s3cmd package from
2) extract it, and you should get a folder called s3cmd-master
3) put the folder to anywhere you want (at here I move it to ~/)
4) create a script called s3cmd under dir /usr/local/bin/ and put the following lines as its content.
~/s3cmd-master/s3cmd $*

5) make the created script executable: sudo chmod +x /usr/local/bin/s3cmd

DONE and you should able to use s3cmd command without specify the full path. 

Friday, June 20, 2014

Install Solr with Tomcat on CentOS

You must have at least version 1.6 in order to run Solr.

Installing Tomcat

yum -y install tomcat6 tomcat6-webapps tomcat6-admin-webapps
chkconfig tomcat6 on
service tomcat6 start
Use a web browser to check it is working correctly.
If you cannot connect, check your firewall rules
iptables -vnL --line-numbers
and make the necessary changes (Check these iptables snippets).

Config Tomcat

1. Tomcat user.

Ok, let’s go on. Add the following lines between the <tomcat-users> tag in /etc/tomcat6/tomcat-users.xml to add a manager role to your server:
<role rolename="manager" />
<role rolename="admin" />
<user username="tomcat" password="Pass.123" roles="manager,admin" />
Important: Make sure to change the previous username and password!

2. Change port to 8983.

Solr will be running on port 8983, so we need to change the default port of Tomcat to 8983 by modify /etc/tomcat6/server.xml and replace all 8080 port to 8983.
Restart Tomcat
service tomcat6 restart

3. Add a rule for port 8983 to iptables

sudo vi /etc/sysconfig/iptables
Add the following line into it.
-A INPUT -m state --state NEW -m tcp -p tcp --dport 8983 -j ACCEPT

Installing Apache Commons Loggins

Download Apache Commons Logging from to your home directory.
Untar your file and copy the required files to your tomcat library folder:
tar zxf commons-logging-1.1.3.tar.gz
cd commons-loggins-1.1.3
cp commons-logging-*.jar /usr/share/tomcat6/lib

Installing SLF4J

Download the right version from to your home directory.
Untar your file and copy the required files to your tomcat library folder:
tar zxf slf4j-1.7.5.tar.gz
cd slf4j-1.7.5
cp slf4j-*.jar /usr/share/tomcat6/lib
Update: Latest versions include a couple Android files (slf4j-android-1.7.7.jar and slf4j-android-1.7.7-sources.jar). Do not copy them. They will be read by tomcat preventing Solr to start.

Installing Solr (Finally!)

Download Solr (version 4.5.1) from the Apache website to your home directory:
Uncompress it:
tar zxf ~/solr-4.5.1.tar.gz
Copy (and rename) the included Solr war file (solr.war) to your Tomcat apps folder:
cp ~/solr-4.5.1/dist/solr-4.5.1.war /usr/share/tomcat6/webapps/solr.war
Now it’s time to create the folder that will keep your Solr index and documents. We also have to copy a basic Solr structure into it. Fortunately, Solr comes with a predefined structure and includes preconfigured files in it. You have to make sure that there is plenty of space wherever you place this folder, since it can grow a lot. By the way, you can place this folder out of your Tomcat webapps folder.
In my case, I have decided to put it in /home/solr. So, let’s do it:
mkdir /home/solr
cp -R ~/solr-4.5.1/example/solr/* /home/solr
chown -R tomcat /home/solr
Restart Tomcat.
service tomcat6 restart
Solr is not ready yet, however this last time we restarted Tomcat it unpackaged the solr.war file. We need to edit a file inside that unpackaged directory structure.
Edit your Solr web.xml file to let it know where your solar directory is located:
nano /usr/share/tomcat6/webapps/solr/WEB-INF/web.xml
Look for the following code and edit it (don’t forget to remove the comment markers!):
Ok… so, restart Tomcat
service tomcat6 restart
On your web browser go to:
If everything is ok, your Solr instance should be ready to go.


I recommend you to completely stop and start Tomcat over and over and take a look at:
tail /var/log/tomcat6/catalina.out
If you do not restart Tomcat each time, errors will not be logged to the catalina.out file.

500 issue

Mostly you will get a error with solr and error message is like "SolrCore ‘collection1′ is not available due to init failure: Could not load config file /home/solr/collection1/solrconfig.xml"
This is because by default solr enable clustering and we can disable it by modifying /home/solr/collection1/conf/solrconfig.xml.
vi /home/solr/collection1/conf/solrconfig.xml
Search for <bool name="clustering">true</bool> and change it to false.
Restart tomcat.

Write permission issue to /home/solr folder

Modify the permission to /home/solr and make sure the user who running tomcat/solr has write permission to it.


Thursday, June 19, 2014

Jenkins Plugins - Claim, Brakeman, Ruby Metrics


This is now installed on Jenkins & will let people claim broken builds - ie tell the world that you are going to fix it and how long you think it will take.
It needs to be enabled on a project by project basis, you do this when configuring your project by adding the 'Allow broken build claiming' post build action.


This is now installed on Jenkins & will display the output of the Brakeman static analysis gem.
To configure:
1 Add the brakeman gem to your Gemfile, eg
group :test do
  gem "brakeman"
2 Add a line at the end of your Jenkins execute shell task to run brakeman, eg
bash -l -c 'brakeman -o brakeman-output.tabs'
3 Enable displaying the output by adding the 'Publish brakeman warnings' post build action to your project Jenkins configuration & setting the output file to point at the file you specified in step 2.

Ruby Metrics

The Jenkins Ruby Metrics plugin will display the output of rcov unit test coverage.
The easiest way we found to set this up:
1. Add simplecov & simplecov-rcov to your Gemfile, eg
group :test do
  gem "simplecov", ">=0.3.8", :require => false
  gem 'simplecov-rcov'
2. Create a file called .simplecov in the root directory of your project
require 'simplecov-rcov'
class SimpleCov::Formatter::MergedFormatter
  def format(result)
SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter
SimpleCov.start 'rails' do
  # any custom configs like groups and filters can be here at a central place
  add_filter "/vendor/"
3. Update spec/spec_helper & features/support/env.rb to require simplecov  ... add this after require 'rubygems' but before all else
require 'simplecov'
4. In Jenkins, enable displaying the output by adding the 'Publish rcov report' post build action in your project Jenkins configuration and pointing it at the coverage/rcov directory
5. See beautiful reports in Jenkins, and also locally when you've run the tests
6. Improve your tests!

Report on Vulnerable Gems with Bundle-Audit

Bundler Audit is a gem that reports on vulnerable gems in your Gemfile. Its similar to Gemnasium or but can more easily be integrated into Jenkins. Its also recommended by the Brakeman people.


Add the following to your Gemfile in the development group:
  gem 'bundler-audit'
bundle install

Run Locally

bundle-audit update
This will output any vulnerable Gem versions you have, or a nice green message if you're ok

Integrate with Jenkins

To display the results on the project home page

Add the following to your "Execute Shell" build step:
bundle-audit update
bundle-audit > bundle-audit.txt
Then under "Post build actions", add "Publish rich text message"
Select "confluence" markup, and paste the following:
h2. Bundle Audit Results
Now re-run your build and the results will display

Take it a step further and make Jenkins fail when there's vulnerable gems

Modify your execute shell build step to check the output of bundle-audit. Here's a simple example script which does this (see SnapDeploy for example)
cd ~
source /var/lib/jenkins/.rvm/scripts/rvm
bundle install
bundle update sqlite3

# Prepare db for testing
RAILS_ENV=test bundle exec rake db:create db:migrate db:test:prepare --trace

# Run rspec
RAILS_ENV=test bundle exec rspec --no-color

# Run cucumber
RAILS_ENV=test bundle exec cucumber -p jenkins

# Run Brakeman
brakeman -o brakeman-output.tabs --no-progress --separate-models

# Run bundle audit
bundle-audit update
bundle-audit > bundle-audit.txt
cat bundle-audit.txt
grep -Fxq "No unpatched versions found" bundle-audit.txt

# Fail if rspec or cucumber failed or there's vulnerable gems
exit "$(($rspec_status + $cucumber_status + $audit_status))"

Thursday, February 13, 2014


IF you are doing Android development on a Mac using a second monitor connected, then you properly will get a error and the emulator is not started.  The error message “emulator64-arm quit unexpectedly”, this is a bug from Android dev tools and has beed filed

The solution is:

cd ~/.android/avd/name_of_the_emulator.avd
vim emulator-user.ini

now change window.x = 0


Friday, January 3, 2014

Host Maven repos on Github

Sometimes, we want to run a project as Maven repo to support other projects OR make it be available to other people to use as Maven dependencies. And I think hosting Maven artifacts on Github is more approachable than running a proper Maven Repository. In this post, I am going to show you how to do it step by step with a example.

1. Set up your Maven repo project on Github.

 You need to figure out where you are going to deploy and then host your project's artifacts. Here I have created a new repo on Github, and checked it out at the root of my dev directory.

dhcp123:server htxiong$ git clone
Initialized empty Git repository in ~/dev/htxiong-mvn-repo/.git/

warning: You appear to have cloned an empty repository.

2. Set up separate snapshots and releases directories.

This is not a technical necessity, but its good to maintain a ongoing project by separating snapshots and releases directories. 

dhcp123:server htxiong$ cd htxiong-mvn-repo
dhcp123:server htxiong$ mkdir snapshots
dhcp123:server htxiong$ mkdir releases

3. Deploy your project's artifacts to Maven repo.

In your project's pom.xml file, there is a configuration element <distributionManagement> that specifies the repositories to which one's project artifacts should be deployed. If you can not find it in your project's pom.xml, please add the following content into it and just before tag </project>.

        <name>Maven repo example Snapshots</name>
        <name>Maven repo example Releases</name>

Now let's build and deploy the project's artifacts by using altDeploymentRepository system property.

For snapshots:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=snapshot-repo::default::file:snapshots clean deploy

For releases:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=repo::default::file:releases clean deploy

And more information about DaltDeploymentRepository  can be found at

4. Push to Github.

In the first 3 steps, we have built and deployed your project's artifacts in our local maven repo which is just like any other git repo, so changes need to be committed and pushed up in order to be useful.

dhcp123:server htxiong$ git add snapshots/*
dhcp123:server htxiong$ git add releases/*
dhcp123:server htxiong$ git commit -m ''add snapshots and releases"
dhcp123:server htxiong$ git push

5. Use your new Maven repo in other projects.

We have already deployed our project on Github as maven repo and the root of this repo will be at, then it is time now to use it in other projects by putting the following content into pom.xml.

For snapshots:


For releases:



Tuesday, August 13, 2013

Set up Shibboleth SP server for AAF integration on CentOS

1. System Requirements
  • Apache is installed.
  • Reverse proxy is set up.
  • SSL is enabled for Apache.
2. Install timer and sync time.
This step is very important, because Shibboleth SP server needs to run on a server which system time is sync with the IDP server. 
$ sudo yum install ntp $ sudo ntpdate server
3. Install & config Shibbloleth
Please refer to for a full document.
Download and installation (as root)
# yum install httpd mod_ssl # wget -P /etc/yum.repos.d # yum install shibboleth
Generate certificate with the correct hostname
Run the following, substituting the externally visible hostname for
# cd /etc/shibboleth # ./ -f -h -e


Download AAF metadata signing certificate

AAF Test Federation
# wget -O /etc/shibboleth/aaf-metadata-cert.pem
AAF Production Federation
# wget -O /etc/shibboleth/aaf-metadata-cert.pem

Edit /etc/shibboleth/shibboleth2.xml

Replace all instances of with your hostname.
  1. In the <Sessions> element, make session handler use SSL: set handlerSSL="true"
    and set cookieProps="https"
  2. change the handlerURL from a relative one ("/Shibboleth.sso" to an absolute one - handlerURL="".
  3. Optionally, customize in the <Errors> element the pages and settings. Your users will come in contact with these if an error occurs. Change the SupportContact attribute to something more meaningful than root@localhost.
  4. Load the federation metadata: add the following (or equivalent) section into shibboleth2.xml just above the sample (commented-out) MetadataProvider element
    AAF Test Federation
    <MetadataProvider type="XML" uri=""
         backingFilePath="metadata.aaf.xml" reloadInterval="7200">
       <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
       <MetadataFilter type="Signature" certificate="aaf-metadata-cert.pem"/>
    AAF Production Federation
    <MetadataProvider type="XML" uri=""
         backingFilePath="metadata.aaf.xml" reloadInterval="7200">
       <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
       <MetadataFilter type="Signature" certificate="aaf-metadata-cert.pem"/>

  5. Locate the <SSO> element (new in SP 2.4) and:
    • Remove reference to default - delete the entityID attribute.
    • Configure the Discovery Service URL in the discoveryURL attribute:
      AAF Test Federation
      AAF Production Federation

Edit /etc/shibboleth/attribute-map.xml

Mapping all attributes you need to request from IDP.

64-bit platforms

On x86_64, if you have installed also the i386 version of shibboleth and its configuration Apache configuration file is taking over, edit/etc/httpd/conf.d/shib.conf and change the path to the Shibboleth Apache module to the 64-bit version:
 LoadModule mod_shib /usr/lib64/shibboleth/


There are 2 different Shibboleth-related log files you can access for troubleshooting.
  • native.log: is located in /var/log/httpd and can be configured in /etc/shibboleth/native.logger
  • shibd.log: is located in /var/log/shibboleth and can be configured in /etc/shibboleth/shibd.logger
  • shibd_warn.log: is located in /var/log/shibboleth
Make sure that the right processes have write permissions to the log files!

Protect a resource

You can protect a resource with Shibboleth by configuring your Apache webserver. Edit the file /etc/httpd/conf.d/shib.conf, you can add as many location to protect by SP server as you want and /secure is a default example:
<Location /secure>
    ShibRequestSetting authType shibboleth
    ShibRequestSetting requireSession true
    require valid-user
More information on how to protect your resource can be found on

Register your SP server in AAF Federation Registry

Before you can use you SP server, you have to register it to AAF Federation Registry
AAF Test Federation
AAF Production Federation

Finishing up

Start up Apache and shibd:
# service httpd start
# service shibd start
# chkconfig httpd on
# chkconfig shibd on


In order to test if everything is working properly, lets create a simple PHP site which will require AAF authentication.
  1. Go to your apache documentRoot (/var/www/html) and create a directory called secure
  2. Create a file called test.php containing the following,
         while (list($var,$value) = each ($_SERVER)) {
               echo "$var => $value <br />";
  3. Edit the httpd.conf file and verify that the php5 module is uncommented.
    LoadModule php5_module libexec/apache2/
  4. Then you can test SP server set up by visiting https://hostname/secure/test.php.


We recommend shibd start before apache start so that in case of shibd errors, apache isn't trying to make erroneous connections while something is being fixed.

Configure Shibboleth to protect Java Servlets

We recommend shibd start before apache start so that in case of shibd errors, apache isn't trying to make erroneous connections while something is being fixed.

The Shibboleth SP is presently only implemented in C++ as a module for Apache httpd, IIS, and NSAPI. However, it's quite easy to use the Shibboleth SP to provide authentication information for Java servlets in a wide variety of servlet containers.
In the setup described here, requests from browsers are intercepted first by Apache httpd. The Shibboleth SP then checks these requests to enforce authentication requirements. After an assertion is received and a Shibboleth session is established, the SP or Apache httpd can enforce access control rules, or it can just pass attributes to the application. The request is then forwarded to the servlet through the use of the AJP13 protocol. Subsequent requests can leverage the Shibboleth session or a session maintained by the application or servlet container to persist the login.

Setup AJP13 support in your servlet container

This step depends on your servlet container.
  • Tomcat: Tomcat has an AJP 1.3 connector enabled by default.
    • Setting the tomcatAuthentication="false" attribute on the AJP <Connector> element allows for passing REMOTE_USER from Apache httpd. See Tomcat's AJP Connector documentation for more.
  • Jetty: Jetty's documentation has good instructions on how to enable both Jetty and your application to listen on AJP 1.3.
Be careful that there is no direct HTTP listener opened by the servlet container. If, for example, there's an HTTP connector listening on port 8080 and no interceding firewall, users would be able to directly access the servlet on port 8080, which bypasses Apache httpd. This also means they would bypass Shibboleth authentication and authorization.

Setup reverse proxy in Apache to redirect hostname/appname to ajp://hostname:8009/appname.

Step by step instruction of this step can be found