Use both optional arguments and positional arguments in Python apps

Normally python scripts accept arguments from command line, and in Python you can achieve that by using ArgumentParser.

e.g. Your Python script named MyApp and it accepts 4 arguments (foo1, foo2, foo3 and foo4) from command line, then your code will be something like:

parser.add_argument(
            '--foo1',
            help="xxxxx"
        )
parser.add_argument(
            '--foo2',
            help="xxxxxx"
        )
parser.add_argument(
            '--foo3',
            help="xxxxxx"
        )
parser.add_argument(
            '--foo4',
            help="xxxxxx"
        )

And all these 4 arguments are optional, if in the case of foo1 is required, you can set required=True.

parser.add_argument(
            '--foo1',
            required=True,
            help="xxxxx"
        )

This works, but its not a good solution (more details can be found https://docs.python.org/3/library/argparse.html#required).

So, set foo1 as positional argument might be better, and you can do like:

parser.add_argument(
            'foo1',
            help="xxxxx"
        )
parser.add_argument(
            '--foo2',
            help="xxxxxx"
        )
parser.add_argument(
            '--foo3',
            help="xxxxxx"
        )
parser.add_argument(
            '--foo4',
            help="xxxxxx"
        )

Then your python script can be ran: python -m MyApp value1 --foo2=value2 --foo3=value3 --foo4=value4.

How to setup a script for you Apps/binary executables which are not installed via command line OR app store

I will use s3cmd as a example to show how to setup a script for your apps/binaries which are not installed via command line Or app store.

1) Download s3cmd package from http://s3tools.org/download

2) extract it, and you should get a folder called s3cmd-master

3) put the folder to anywhere you want (at here I move it to ~/)

4) create a script called s3cmd under dir /usr/local/bin/ and put the following lines as its content.

#!/bin/sh

~/s3cmd-master/s3cmd $*

5) make the created script executable: sudo chmod +x /usr/local/bin/s3cmd

DONE and you should able to use s3cmd command without specify the full path. 

Install Solr with Tomcat on CentOS

You must have at least version 1.6 in order to run Solr.

Installing Tomcat

yum -y install tomcat6 tomcat6-webapps tomcat6-admin-webapps
chkconfig tomcat6 on
service tomcat6 start

Use a web browser to check it is working correctly.

http://localhost:8080/

If you cannot connect, check your firewall rules

iptables -vnL --line-numbers

and make the necessary changes (Check these iptables snippets).

Config Tomcat

1. Tomcat user.

Ok, let’s go on. Add the following lines between the <tomcat-users> tag in /etc/tomcat6/tomcat-users.xml to add a manager role to your server:

<role rolename="manager" />
<role rolename="admin" />
<user username="tomcat" password="Pass.123" roles="manager,admin" />

Important: Make sure to change the previous username and password!

2. Change port to 8983.

Solr will be running on port 8983, so we need to change the default port of Tomcat to 8983 by modify /etc/tomcat6/server.xml and replace all 8080 port to 8983.

Restart Tomcat

service tomcat6 restart

3. Add a rule for port 8983 to iptables

sudo vi /etc/sysconfig/iptables

Add the following line into it.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8983 -j ACCEPT

Installing Apache Commons Loggins

Download Apache Commons Logging from http://commons.apache.org/proper/commons-logging/download_logging.cgi to your home directory.

Untar your file and copy the required files to your tomcat library folder:

cd
tar zxf commons-logging-1.1.3.tar.gz
cd commons-loggins-1.1.3
cp commons-logging-*.jar /usr/share/tomcat6/lib

Installing SLF4J

Download the right version from http://www.slf4j.org/ to your home directory.

Untar your file and copy the required files to your tomcat library folder:

cd
tar zxf slf4j-1.7.5.tar.gz
cd slf4j-1.7.5
cp slf4j-*.jar /usr/share/tomcat6/lib

Update: Latest versions include a couple Android files (slf4j-android-1.7.7.jar and slf4j-android-1.7.7-sources.jar). Do not copy them. They will be read by tomcat preventing Solr to start.

Installing Solr (Finally!)

Download Solr (version 4.5.1) from the Apache website to your home directory:

http://lucene.apache.org/solr/downloads.html

Uncompress it:

cd
tar zxf ~/solr-4.5.1.tar.gz

Copy (and rename) the included Solr war file (solr.war) to your Tomcat apps folder:

cp ~/solr-4.5.1/dist/solr-4.5.1.war /usr/share/tomcat6/webapps/solr.war

Now it’s time to create the folder that will keep your Solr index and documents. We also have to copy a basic Solr structure into it. Fortunately, Solr comes with a predefined structure and includes preconfigured files in it. You have to make sure that there is plenty of space wherever you place this folder, since it can grow a lot. By the way, you can place this folder out of your Tomcat webapps folder.

In my case, I have decided to put it in /home/solr. So, let’s do it:

mkdir /home/solr
cp -R ~/solr-4.5.1/example/solr/* /home/solr
chown -R tomcat /home/solr

Restart Tomcat.

service tomcat6 restart

Solr is not ready yet, however this last time we restarted Tomcat it unpackaged the solr.war file. We need to edit a file inside that unpackaged directory structure.

Edit your Solr web.xml file to let it know where your solar directory is located:

nano /usr/share/tomcat6/webapps/solr/WEB-INF/web.xml

Look for the following code and edit it (don’t forget to remove the comment markers!):

<env-entry>
    <env-entry-name>solr/home</env-entry-name>
    <env-entry-value>/home/solr</env-entry-value>
    <env-entry-type>java.lang.String</env-entry-type>
</env-entry>

Ok… so, restart Tomcat

service tomcat6 restart

On your web browser go to:

http://localhost:8983/solr

If everything is ok, your Solr instance should be ready to go.

Troubleshooting

I recommend you to completely stop and start Tomcat over and over and take a look at:

tail /var/log/tomcat6/catalina.out

If you do not restart Tomcat each time, errors will not be logged to the catalina.out file.

500 issue

Mostly you will get a error with solr and error message is like "SolrCore ‘collection1′ is not available due to init failure: Could not load config file /home/solr/collection1/solrconfig.xml"

This is because by default solr enable clustering and we can disable it by modifying /home/solr/collection1/conf/solrconfig.xml.

vi /home/solr/collection1/conf/solrconfig.xml

Search for <bool name="clustering">true</bool> and change it to false.

Restart tomcat.

Write permission issue to /home/solr folder

Modify the permission to /home/solr and make sure the user who running tomcat/solr has write permission to it.

References

http://andres.jaimes.net/878/setup-lucene-solr-centos-tomcat/

https://wiki.apache.org/solr/SolrLogging#Using_the_example_logging_setup_in_containers_other_than_Jetty

http://wiki.apache.org/solr/ClusteringComponent

Jenkins Plugins - Claim, Brakeman, Ruby Metrics

Claim

This is now installed on Jenkins & will let people claim broken builds - ie tell the world that you are going to fix it and how long you think it will take.

It needs to be enabled on a project by project basis, you do this when configuring your project by adding the 'Allow broken build claiming' post build action.

Brakeman

This is now installed on Jenkins & will display the output of the Brakeman static analysis gem.

To configure:

1 Add the brakeman gem to your Gemfile, eg

group :test do ... gem "brakeman" ... end

2 Add a line at the end of your Jenkins execute shell task to run brakeman, eg

bash -l -c 'brakeman -o brakeman-output.tabs'

3 Enable displaying the output by adding the 'Publish brakeman warnings' post build action to your project Jenkins configuration & setting the output file to point at the file you specified in step 2.

Ruby Metrics

The Jenkins Ruby Metrics plugin will display the output of rcov unit test coverage.

The easiest way we found to set this up:

1. Add simplecov & simplecov-rcov to your Gemfile, eg

group :test do ... gem "simplecov", ">=0.3.8", :require => false gem 'simplecov-rcov' ... end

2. Create a file called .simplecov in the root directory of your project

require 'simplecov-rcov' class SimpleCov::Formatter::MergedFormatter def format(result) SimpleCov::Formatter::HTMLFormatter.new.format(result) SimpleCov::Formatter::RcovFormatter.new.format(result) end end SimpleCov.formatter = SimpleCov::Formatter::MergedFormatter SimpleCov.start 'rails' do # any custom configs like groups and filters can be here at a central place add_filter "/vendor/" end

3. Update spec/spec_helper & features/support/env.rb to require simplecov  ... add this after require 'rubygems' but before all else

require 'simplecov'

4. In Jenkins, enable displaying the output by adding the 'Publish rcov report' post build action in your project Jenkins configuration and pointing it at the coverage/rcov directory

5. See beautiful reports in Jenkins, and also locally when you've run the tests

6. Improve your tests!

.

Report on Vulnerable Gems with Bundle-Audit

Bundler Audit is a gem that reports on vulnerable gems in your Gemfile. Its similar to Gemnasium or https://hakiri.io/facets but can more easily be integrated into Jenkins. Its also recommended by the Brakeman people.

Install

Add the following to your Gemfile in the development group:

gem 'bundler-audit'

Run

bundle install

Run Locally

Run

bundle-audit update bundle-audit

This will output any vulnerable Gem versions you have, or a nice green message if you're ok

Integrate with Jenkins

To display the results on the project home page

Add the following to your "Execute Shell" build step:

bundle-audit update bundle-audit > bundle-audit.txt

Then under "Post build actions", add "Publish rich text message"

Select "confluence" markup, and paste the following:

h2. Bundle Audit Results ${FILE:bundle-audit.txt}

Now re-run your build and the results will display

Take it a step further and make Jenkins fail when there's vulnerable gems

Modify your execute shell build step to check the output of bundle-audit. Here's a simple example script which does this (see SnapDeploy for example)

#!/bin/bash cd ~ source /var/lib/jenkins/.rvm/scripts/rvm cd $WORKSPACE bundle install bundle update sqlite3 # Prepare db for testing RAILS_ENV=test bundle exec rake db:create db:migrate db:test:prepare --trace # Run rspec RAILS_ENV=test bundle exec rspec --no-color rspec_status=$? # Run cucumber RAILS_ENV=test bundle exec cucumber -p jenkins cucumber_status=$? # Run Brakeman brakeman -o brakeman-output.tabs --no-progress --separate-models # Run bundle audit bundle-audit update bundle-audit > bundle-audit.txt echo "BUNDLE AUDIT RESULTS:" cat bundle-audit.txt grep -Fxq "No unpatched versions found" bundle-audit.txt audit_status=$? # Fail if rspec or cucumber failed or there's vulnerable gems exit "$(($rspec_status + $cucumber_status + $audit_status))"

AVD CRASHES ON LAUNCH (EMULATOR64-ARM QUIT UNEXPECTEDLY)

IF you are doing Android development on a Mac using a second monitor connected, then you properly will get a error and the emulator is not started.  The error message “emulator64-arm quit unexpectedly”, this is a bug from Android dev tools and has beed filed https://code.google.com/p/android/issues/detail?id=38371.

The solution is:

cd ~/.android/avd/name_of_the_emulator.avd

vim emulator-user.ini

now change window.x = 0

DONE.

Host Maven repos on Github

Sometimes, we want to run a project as Maven repo to support other projects OR make it be available to other people to use as Maven dependencies. And I think hosting Maven artifacts on Github is more approachable than running a proper Maven Repository. In this post, I am going to show you how to do it step by step with a example.

1. Set up your Maven repo project on Github.

 You need to figure out where you are going to deploy and then host your project's artifacts. Here I have created a new repo on Github, and checked it out at the root of my dev directory.

dhcp123:server htxiong$ git clone git@github.com:htxiong/htxiong-mvn-repo.git
Initialized empty Git repository in ~/dev/htxiong-mvn-repo/.git/

warning: You appear to have cloned an empty repository.

2. Set up separate snapshots and releases directories.

This is not a technical necessity, but its good to maintain a ongoing project by separating snapshots and releases directories. 

dhcp123:server htxiong$ cd htxiong-mvn-repo
dhcp123:server htxiong$ mkdir snapshots
dhcp123:server htxiong$ mkdir releases

3. Deploy your project's artifacts to Maven repo.

In your project's pom.xml file, there is a configuration element <distributionManagement> that specifies the repositories to which one's project artifacts should be deployed. If you can not find it in your project's pom.xml, please add the following content into it and just before tag </project>.

<distributionManagement>

    <snapshotRepository>

        <id>mvn-repo-snapshot</id>

        <name>Maven repo example Snapshots</name>

         <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/snapshots</url>

    </snapshotRepository>

    <repository>

        <id>mvn-repo</id>

        <name>Maven repo example Releases</name>

        <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/releases</url>

    </repository>

</distributionManagement> 

Now let's build and deploy the project's artifacts by using altDeploymentRepository system property.

For snapshots:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=snapshot-repo::default::file:snapshots clean deploy

For releases:
dhcp123:server htxiong$ mvn -DaltDeploymentRepository=repo::default::file:releases clean deploy

And more information about DaltDeploymentRepository  can be found at http://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html#altDeploymentRepository.

4. Push to Github.

In the first 3 steps, we have built and deployed your project's artifacts in our local maven repo which is just like any other git repo, so changes need to be committed and pushed up in order to be useful.

dhcp123:server htxiong$ git add snapshots/*
dhcp123:server htxiong$ git add releases/*
dhcp123:server htxiong$ git commit -m ''add snapshots and releases"
dhcp123:server htxiong$ git push

5. Use your new Maven repo in other projects.

We have already deployed our project on Github as maven repo and the root of this repo will be at https://github.com/htxiong/htxiong-mvn-repo/raw/master/, then it is time now to use it in other projects by putting the following content into pom.xml.

For snapshots:

<repository>

    <id>-htxiong-mvn-repo</id>

    <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/snapshots</url>

</repository>

For releases:

<repository>

    <id>-htxiong-mvn-repo</id>

    <url>https://github.com/htxiong/htxiong-mvn-repo/raw/master/releases</url>

</repository>

DONE!

Set up Shibboleth SP server for AAF integration on CentOS

1. System Requirements

• Apache is installed.
• Reverse proxy is set up.
• SSL is enabled for Apache.

2. Install timer and sync time.

This step is very important, because Shibboleth SP server needs to run on a server which system time is sync with the IDP server. 

 

$ sudo yum install ntp $ sudo ntpdate server 0.centos.pool.ntp.org

3. Install & config Shibbloleth

Please refer to http://wiki.aaf.edu.au/tech-info/sp-install-guide for a full document.

Download and installation (as root)

# yum install httpd mod_ssl # wget http://download.opensuse.org/repositories/security://shibboleth/CentOS_CentOS-6/security:shibboleth.repo -P /etc/yum.repos.d # yum install shibboleth

Generate certificate with the correct hostname

Run the following, substituting the externally visible hostname for sp.example.org:

# cd /etc/shibboleth # ./keygen.sh -f -h sp.example.org -e https://sp.example.org/shibboleth

Configuration

Download AAF metadata signing certificate

AAF Test Federation

# wget https://ds.test.aaf.edu.au/distribution/metadata/aaf-metadata-cert.pem -O /etc/shibboleth/aaf-metadata-cert.pem

AAF Production Federation

# wget https://ds.aaf.edu.au/distribution/metadata/aaf-metadata-cert.pem -O /etc/shibboleth/aaf-metadata-cert.pem

Edit /etc/shibboleth/shibboleth2.xml

Replace all instances of sp.example.org with your hostname.

1. In the <Sessions> element, make session handler use SSL: set handlerSSL="true"
   and set cookieProps="https"
2. change the handlerURL from a relative one ("/Shibboleth.sso" to an absolute one - handlerURL="https://sp.example.org/Shibboleth.sso".
3. Optionally, customize in the <Errors> element the pages and settings. Your users will come in contact with these if an error occurs. Change the SupportContact attribute to something more meaningful than root@localhost.
4. Load the federation metadata: add the following (or equivalent) section into shibboleth2.xml just above the sample (commented-out) MetadataProvider element
   AAF Test Federation

   <MetadataProvider type="XML" uri="https://ds.test.aaf.edu.au/distribution/metadata/metadata.aaf.signed.complete.xml"
        backingFilePath="metadata.aaf.xml" reloadInterval="7200">
      <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
      <MetadataFilter type="Signature" certificate="aaf-metadata-cert.pem"/>
   </MetadataProvider>

   AAF Production Federation

   <MetadataProvider type="XML" uri="https://ds.aaf.edu.au/distribution/metadata/metadata.aaf.signed.complete.xml"
        backingFilePath="metadata.aaf.xml" reloadInterval="7200">
      <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
      <MetadataFilter type="Signature" certificate="aaf-metadata-cert.pem"/>
   </MetadataProvider>

5. Locate the <SSO> element (new in SP 2.4) and:
   • Remove reference to default idp.example.org - delete the entityID attribute.
   • Configure the Discovery Service URL in the discoveryURL attribute:
     
     AAF Test Federation

     discoveryURL="https://ds.test.aaf.edu.au/discovery/DS"

     AAF Production Federation

     discoveryURL="https://ds.aaf.edu.au/discovery/DS"

    

Edit /etc/shibboleth/attribute-map.xml

Mapping all attributes you need to request from IDP.

64-bit platforms

On x86_64, if you have installed also the i386 version of shibboleth and its configuration Apache configuration file is taking over, edit/etc/httpd/conf.d/shib.conf and change the path to the Shibboleth Apache module to the 64-bit version:

 LoadModule mod_shib /usr/lib64/shibboleth/mod_shib_22.so

Logging

There are 2 different Shibboleth-related log files you can access for troubleshooting.

• native.log: is located in /var/log/httpd and can be configured in /etc/shibboleth/native.logger
• shibd.log: is located in /var/log/shibboleth and can be configured in /etc/shibboleth/shibd.logger
• shibd_warn.log: is located in /var/log/shibboleth

Make sure that the right processes have write permissions to the log files!

Protect a resource

You can protect a resource with Shibboleth by configuring your Apache webserver. Edit the file /etc/httpd/conf.d/shib.conf, you can add as many location to protect by SP server as you want and /secure is a default example:

<Location /secure>
    ShibRequestSetting authType shibboleth
    ShibRequestSetting requireSession true
    require valid-user
</Location>

More information on how to protect your resource can be found onhttps://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPhtaccess.

Register your SP server in AAF Federation Registry

Before you can use you SP server, you have to register it to AAF Federation Registry

AAF Test Federation

https://manager.test.aaf.edu.au/federationregistry/

AAF Production Federation

https://manager.aaf.edu.au/federationregistry/

Finishing up

Start up Apache and shibd:

# service httpd start
# service shibd start
# chkconfig httpd on
# chkconfig shibd on

Testing

In order to test if everything is working properly, lets create a simple PHP site which will require AAF authentication.

1. Go to your apache documentRoot (/var/www/html) and create a directory called secure
2. Create a file called test.php containing the following,

   SECURE: -------------- <?php      while (list($var,$value) = each ($_SERVER)) {            echo "$var => $value <br />";      } ?>

3. Edit the httpd.conf file and verify that the php5 module is uncommented.

   LoadModule php5_module libexec/apache2/libphp5.so

4. Then you can test SP server set up by visiting https://hostname/secure/test.php.

Startup

We recommend shibd start before apache start so that in case of shibd errors, apache isn't trying to make erroneous connections while something is being fixed.

Configure Shibboleth to protect Java Servlets

We recommend shibd start before apache start so that in case of shibd errors, apache isn't trying to make erroneous connections while something is being fixed.

The Shibboleth SP is presently only implemented in C++ as a module for Apache httpd, IIS, and NSAPI. However, it's quite easy to use the Shibboleth SP to provide authentication information for Java servlets in a wide variety of servlet containers.

In the setup described here, requests from browsers are intercepted first by Apache httpd. The Shibboleth SP then checks these requests to enforce authentication requirements. After an assertion is received and a Shibboleth session is established, the SP or Apache httpd can enforce access control rules, or it can just pass attributes to the application. The request is then forwarded to the servlet through the use of the AJP13 protocol. Subsequent requests can leverage the Shibboleth session or a session maintained by the application or servlet container to persist the login.

Setup AJP13 support in your servlet container

This step depends on your servlet container.

• Tomcat: Tomcat has an AJP 1.3 connector enabled by default.
  • Setting the tomcatAuthentication="false" attribute on the AJP <Connector> element allows for passing REMOTE_USER from Apache httpd. See Tomcat's AJP Connector documentation for more.
• Jetty: Jetty's documentation has good instructions on how to enable both Jetty and your application to listen on AJP 1.3.

Be careful that there is no direct HTTP listener opened by the servlet container. If, for example, there's an HTTP connector listening on port 8080 and no interceding firewall, users would be able to directly access the servlet on port 8080, which bypasses Apache httpd. This also means they would bypass Shibboleth authentication and authorization.

Setup reverse proxy in Apache to redirect hostname/appname to ajp://hostname:8009/appname.

Step by step instruction of this step can be found http://blog.htxiong.com/2013/08/force-https-for-entire-serverdomain.html

DONE!

Reference

https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPJavaInstall

https://github.com/IntersectAustralia/aodn-portal/wiki/The-First-Time-Deployment-Guide#15-install-and-configure-apache-and-tomcat-integration-vm1-vm2-vm4   

http://wiki.aaf.edu.au/tech-info/sp-install-guide