Getting the output from a sub-process

A few times recently i’ve needed to capture the output from an OS specific command in Java.   It’s not terribly difficult, but this little example may save you some time.

package test;

import java.io.BufferedReader;
import java.io.InputStreamReader;

public class Main {
    private final static String COMMAND = "/usr/bin/uptime";

    public static void main(String[] args) throws Exception {

       Process process = Runtime.getRuntime().exec(COMMAND);
       BufferedReader stdout =
            new BufferedReader(new InputStreamReader(process.getInputStream()));
       String line = "";
       while ((line=stdout.readLine())!=null) {
          System.out.println(line);
       }
       process.waitFor();  // unlikely we'd ever wait as we've got EOF on stdout
       System.out.println("Exit code = "+process.exitValue());
    }
}

And here is the output as you would expect :

 08:30:47 up 12 days, 23:53,  1 user,  load average: 1.87, 1.99, 2.07
Exit code = 0

The empire strikes back…. or does it?

I still chuckle from the completely non-scientific way of benchmarking Java versus Python in a previous posting. Anyway, to further rub salt into the wounds we can have a look at the results from the Debian language shootout .   Maybe I should just present the results without commentary …. nah where would be the fun in that

The graph is indicating that yes, java outperforms python in just about all of the benchmarks.  Sure the memory usage of the java application is higher and I concede that that is one of the major drawbacks of the java platform.  If the bug discussed in this prior post was actually fixed then there would be little to whinge about – but i’m sure people would find something, because people love whinging, me included 

Interestingly here are the results of a Java versus Perl comparison as well.

What! Java is faster than perl, how can this be ! /me removes tongue from cheek.

And for those who still tell me that php is faster than Java…. well let’s see….

Oh dear, poor php.

This sort of data all ways makes me chuckle.  The only bit that makes me uncomfortable is that I didn’t run the tests myself and I haven’t looked at the code contained in the benchmarks.   This is something that I want to do agains the openjdk code base as i’m doing research on the hotspot compiler and this code provides for an interesting test bed.

The 20 cent summary is if you think Python is faster than Java then you really need to have a look at the data.  If you want Python to be faster than Java then analyze the code, explore the Python code base and see if you can make it better – take that as a challenge as everyone benefits from the results.

Yes, I use Python as well as Perl and Java, make them all quicker – see if I care

System.exit(0) Universe Terminated

I’m always interested in cool things running Java, mostly because i’m a nerd, but also because it annoys people when I tell them how cool Java is and let’s face it – that’s a fair bit of fun in itself

Anyway,  the Large Hadron Collider uses Linux, Java and other platforms with both Linux and Java playing a major role in the control systems.   So, how does Java lead to the end of the universe. Well, of course that is a ‘tongue-in-cheek’ view of the legal challenge raised against the LHC where it is speculated that the LHC could create a variety of conditions leading to the end of us all (read more over at wikipedia).   Let’s hope no one leaves a stray System.exit() in the code and accidentally destroy the universe – that really would be a poor programming practice 

The list of really cool practical applications of Java and Linux technology continues to grow.  It seems to me that the LHC is possibly the biggest machine ever created – are there others bigger?

building netbeans from source on ubuntu

Everyone needs a hobby.   Clearly I don’t have one.

I’ve decided that I want the latest and greatest set of features (and bugs) for netbeans so i’m installing netbeans from the code repository and building it on my currently preferred platform (ubuntu 8.04 – yes I know it’s a beta which is only more proof that I need a hobby).

I prefer to do all my bleeding edge test builds on the ’server’ version of ubuntu – why?  I like small footprint build systems that only have the required packages to satisfy my build.  I guess I just don’t like packages sitting there looking at me and this way I know exactly what’s required to make it work on my primary desktop system later.  ie. I can do everything in an experimental manner in the VM and develop detailed documentation of what is required to implement it on other systems.

So, going from the top, I’ve created a brand new hardy server as a VM so that I have a clear view of everything that is required to get the job done.   If your system already has some of the pre-req software loaded then great.

netbeans uses mercurial so to get the source code you need that installed

~$ sudo apt-get install mercurial
~$ mkdir netbeans && cd netbeans
~/netbeans$ hg clone http://hg.netbeans.org/main/
destination directory: main
requesting all changes
adding changesets
adding manifests
adding file changes
added 78686 changesets with 365473 changes to 80849 files
77968 files updated, 0 files merged, 0 files removed, 0 files unresolved

note: to build netbeans we need a few tools, first of them is ant which will drag on a number of dependencies.

sudo apt-get install ant ant-optional
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following extra packages will be installed:
.
.
.

and given that there are issues using gcj  to build netbeans (and I really can’t be bothered looking closer to tell you the truth) I suggest not bothering and just install the sun jdk (note: java5 is required and anything higher wont work at the time of writing this).  Also, don’t forget the ant-optional otherwise you will get complaints about regexp during compilation.

~netbeans$ sudo apt-get install sun-java5-jdk

Add the following to your ~/.nbbuild.properties file

nbjdk.home=/usr/lib/jvm/java-1.5.0-sun
build.compiler=extJavac
javac.compilerargs=-J-Xmx512m

and the following to your ~/.antrc file

ANT_OPTS=-Xmx384m

~/netbeans$ ant -f main/build.xml

and hey presto netbeans should be built (it will take some time).   Now as I’m ssh -X into this server I also need to ensure the xauth package is installed

~/netbeans$ sudo apt-get install xauth

and then I can try out the newly compiled netbeans using

~/netbeans$ ant -f main/build.xml tryme

which should see the netbeans splash screen appear and I can start using the latest development version of netbeans.   Of course, this means you have to deal with any bugs, performance problems and mysterious quirks yourself.   Join the mailing lists and enjoy.

Now that I have a very good understanding of what is required to get netbeans built from source I am confident that doing so will not interfere with anything else running on my primary development system.  Using the results of the above exploration it should be a simple matter of matching the package requirements, rsync-ing over the downloaded repository and you’re good to go.

watching junk with java

Ok,  I had to admit this press release did catch my eye.  To quote :

“The Eglin Space Surveillance Radar (FPS-85), which schedules and tracks catalogued space objects, is currently undergoing a complex modernization process to replace legacy mission-critical components built from one-of-a-kind equipment and custom software. The Sun Java Real-Time System, a high-level development platform for creating applications that require unprecedented execution predictability, will enable ITT to run the new solution using Java technology on Solaris 10 OS and standard hardware.”

As a Linux nerd it’s a shame that it’s Solaris 10 and not one of the RT variants of Linux but any technology that controls a 1250 ton radar array that can track basketball sized objects 22000 nautical miles away is pretty cool in my book

It is disappointing that the Sun Java Real-Time system isn’t opensource, but it least it’s good to note that it is available for real time linux variants such as SLERT.

Oh java…. what are you doing?

Lots of people complain about java performance (and you know who you are )

But if you ask them if they’ve actually looked to see what their application is doing many say ‘nope’.  This is of course frustrating for everyone.

Mostly this happens because people just don’t know how to investigate how their application is performing and in the past it’s been difficult to do.  It’s also possible that they’d rather just complain rather than fixing things – much like me previously whinging about the shortcomings of HTML5

However, if you’re running a  JDK (1.5.0+) which implements the Java Management Extensions then you can use tools such as jconsole or jstat to gather information about the execution profile of your application with a goal to correcting any anti-social behaviours.

Let me give you an example of using these tools on a well known Java application – Tomcat.To utilize these tools you have to turn on the management interface which allows the application VM to be monitored from the local network or across the network ( yes, you can secure the management connection )

So, modifying the Tomcat startup to include -Dcom.sun.management.jmxremote – start up Tomcat and then you can use jconsole <pid>  to get a look at what the JVM is doing for your application.

Attached are some pictures. Of course now that you have some better diagnostic data you can try to understand what may be causing it, possibly tune it better, produce better bug reports for the developers or even use them to assist you in fixing the application yourself – it’s ok, you can still say Java sucks but at least now you’ll have some evidence to support you.

jconsole window showing some performance statistics.

Some basic performance statistics.

Of course, if you want to collect trending information over a longer period then make sure you have a look at jstat.

Just the JVM m’aam

Previously I was looking at Java versus Python performance (here) and I was quite surprised (well not surprised, but more intruiged) by the JVM startup speed so I thought i’d have a bit more of a look to satisfy my curiosity. I don’t have all the answers so don’t hassle me, but what I started doing was using the JNI to create a JVM from a simple C command as opposed to using the java command to do the work and invoke the classloader etc. Anyway, this is how you do it in case you’re suitably bored and want to have a go

Here is the ‘c’ code which is a slightly modified version from the JNI reference manual:


#include <jni.h>
#include <stdlib.h>

#define USER_CLASSPATH “.” /* where to find Prog.class */

int main() {

JNIEnv *env;
JavaVM *jvm;
jint res;
jclass cls;
jmethodID mid;
jclass stringClass;
jobjectArray args;

JavaVMInitArgs vm_args;
JavaVMOption options[1];
options[0].optionString =
“-Djava.class.path=” USER_CLASSPATH;
vm_args.version = JNI_VERSION_1_6;
vm_args.options = options;
vm_args.nOptions = 1;
vm_args.ignoreUnrecognized = JNI_TRUE;
/* Create the Java VM */
res = JNI_CreateJavaVM(&jvm, (void**)&env, &vm_args);

if (res < 0) {
fprintf(stderr, “Can’t create Java VM\n”);
exit(1);
}
exit(0); /* here */
cls = (*env)->FindClass(env, “Prog”);
if (cls == NULL) {
goto destroy;
}

mid = (*env)->GetStaticMethodID(env, cls, “main”,
“([Ljava/lang/String;)V");
if (mid == NULL) {
goto destroy;
}
args = NULL;
(*env)->CallStaticVoidMethod(env, cls, mid, args);

destroy:
if ((*env)->ExceptionOccurred(env)) {
(*env)->ExceptionDescribe(env);
}
(*jvm)->DestroyJavaVM(jvm);
}
Additionally here is the Java code for the class executed by the JVM when instantiated.

public class Prog {
public static void main(String[] args) {
System.out.println(“Hello World”);
}
}


We wont talk about how wordpress has formatted the above - oh well you get the idea

Get things compiled and linked

javac Prog.java
gcc -I /usr/local/jdk1.7.0/include/ -I /usr/local/jdk1.7.0/include/linux/ -L /usr/local/jdk1.7.0/jre/lib/i386 -L /usr/local/jdk1.7.0/jre/lib/i386/client -ljava -ljvm -lverify -o invoke invoke.c

noting that i'm using an openjdk code drop

export LD_LIBRARY_PATH=/usr/local/jdk1.7.0/jre/lib/i386:/usr/local/jdk1.7.0/jre/lib/i386/client

and you should be able to ./invoke to check it out. If you review the invoke.c code above you'll see an exit(0) statement with comment 'here'. If you leave that in and time the execution you will see that the JVM instantiation isn't that bad which is more in line with the empty python example mentioned in the previous blog entry. Removing the exit should see the JVM instantiated and the class file executed.

Interestingly just such a test reveals a far better comparion for VM instantiation with nothing to execute than the previous test. In this case the ratio now becomes :

Test	Ratio (Java/Python)
Interpreter Init	1.6

Java vs Python performance

Always good to provide some non-scientific comparative performance data – tends to rub people the wrong way, but here goes anyway

Anyway, http://www.twistedmatrix.com/~glyph/rant/python-vs-java.html detailed a quick performance evaluation that I thought was ’sort of ‘ useful for some research I was doing – then I saw the date on the page – 2000 – which is shall we say, a ‘tad’ old. So I thought i’d re-run the tests on a current platform, in this case Ubuntu 7.10 ‘Gutsy’. As i’m only interested in the performance ratio between Java and Python rather than the overall performance increase due to improvements in underlying hardware i’ve converted the results on the previous web page to a ratio (Java/Python) as well to see how the systems have changed in the last 7 years.

Test	Ratio (Java/Python) in 2000
Standard Output	4.54
Hashtable	2.06
I/O	1.19
List	0.31
Interpreter Init	6.25
Object Allocation	0.11
Interpreter Speed	0.18

As you can see from those numbers (>1) that Java was considerably slower than Python in the arbitrary tests performed. As I was a little curious to see it for myself I re-ran the code detailed on the page on my fresh gutsy system albeit with an openjdk code drop just to see if things had changed and how.

Test	Ratio (Java/Python) in 2007
Standard Output	2.20
Hashtable	0.64
I/O	0.38
List	0.42
Interpreter Init	12
Object Allocation	0.034
Interpreter Speed	0.86

As you can see Java has improved in most of the tests so that it now significantly outperforms python. I guess the major disappointment is the Interpreter initialization which seems to have gotten significantly worse but overall Java performance has according to these tests improved relative to python.

Please note that I haven’t looked at the suitability of the tests themselves, merely re-run them for myself to try and improve my understanding – i’ve released it here so that there is at least a recent counterpoint to the 2000 article. Flames >/dev/null please

Crypto in OpenJDK

Well we have yet another milestone in the OpenJDK world.   b21 of openjdk now sees the cryptographic code (aka JCE) added to the OpenJDK source tree removing yet another part of the encumbered code (aka binary plugs).    This is a great step and the code is probably well worth a look if you’re into java based crypto as there’s always something to learn in that area.   For more information download drop b21 from openjdk.java.net

Is Java anti-social?

Many people complain that Java applications are anti-social when it comes to the overall operation and management of the system on which they execute. Do Java applications get along with the other kids in the playground? The answer is Yes and No. Yes they can depending on the application, how the application works and more importantly how the system is configured and No they don’t when you expect them to behave like applications written in C.

The most vocal complaints about Java applications revolve around suggestions that Java applications are bloated from a memory utilization perspective. So, why is it this way? It’s really not easy to find accurate definitive answers to this if you’re searching the web, however the root cause seems to be that the JVM doesn’t return memory from the heap back to the operating system even after garbage collection has occurred. This would be a requirement in some cases, such as an embedded JVM that may not have an underlying OS to return memory to, however with a sophisticated OS supporting the JVM I think memory should be returned to allow the OS more options in application workload management.

Admittedly there are some posts around that suggest that there have been improvements in this area but the current position is best illustrated by the following bug report . It’s particularly interesting to note the workaround which I personally don’t find inspirational and it certainly doesn’t consider all of the possible problems with all OS’s if you follow the advice blindly

The RFE is still open and has been since 2002 – again not awe inspiring. Now that Java has been open sourced hopefully we’ll see a JVM that releases memory back to the OS rather than relying on OS based working set management – then maybe java wont get so many bloody noses when it ventures out into the playground.

Does this apply to all JVM’s? How much improvement has there actually been since 2002 and is it quantifiable? I don’t know – still looking.