Mostly Harmless

A long time ago, I wrote how bad x64 Java is. Well the good news is -- no more. With 6u14 just around the corner, sanity will be back. Starting update 14, the 64-bit version with Heap sizes below 32GB will use almost the same amount of memory as the 32-bit JVM. All this because of the use compressed pointers to save memory.

It gets even better, the compressed pointers are already available with the Java SE Performance Release since July 2008 (someone should have told me earlier). All that needs to be done is, use -XX:+UseCompressedOops. So I finally decided to give it a try and compare memory usages.

To compare, I used the following test program with 1.6.0_11i686 (32-bit), 1.6.0_11x64 (64-bit), and 1.6.0_06performace with -XX:+UseCompressedOops (64-bit performance) on my desktop with 8GB memory and Intel quad core processor. The minimum amount of memory required to run the program are reported below.

#File: JavaMemoryProblems.java
import java.util.HashMap;
import java.util.Map;
public class JavaMemoryProblems {
   public static void main(String[] args) {
       Map<String, Integer> map = new HashMap<String, Integer>();
       int max = 1000000;
       for (int i=0; i<max; i++) {
           map.put("key"+i, new Integer(i));
       }
       System.out.println("Finished with a map of size "+map.size());
   }
}

The test program is the same as my previous post, and the decrease in the memory is quite amazing. Time to throw the 32-bit JVM out of the window. I haven't done any tests around increase in CPU time, but I do not expect any penalty in run time based on few of my google searches

Update on 31st May 2009: After using the performance release in production for a few days I realize it is broken. The application is performing weirdly, giving incorrect output and throwing exceptions. I have now switched back to the regular release 6u13 release, and everything is fine again. I am not sure if it is really a Java bug or something else, but I am not taking any chances.

Labels: java, performance

# posted by nileshbansal @ 1:03 am 3 comments - -   -bookmark at -del.icio.us   -

Java is by no means a succinct language. For simple operations, the programmer is expected to punch in a hundred keys; but there are a few little shortcuts that can help (and of course Eclipse is always there to help). One such trick is inline initialization of Map which I stumbled across recently and found to be very interesting.

We are all used to using the following code to initialize a map -

Map<String, String> map = new HashMap<String, String>();
map.put("Harry", "Potter");
map.put("Ron", "Weasley");
map.put("Hermione", "Granger");

The problem with this code is it is four different statements. Using a small static initialization trick, you can make it to be a single statement as follows:

Map<String, String> map = new HashMap<String, String>() {{
   put("Harry", "Potter");
   put("Ron", "Weasley");
   put("Hermione", "Granger");
}};

All we are doing here is sub-classing the HashMap class to an anonymous class, and then using the non-static initialization block to call the put() method three times.

PS - I have not updated this blog in a while, as I have been rather busy lately. From now on I will try to find some time, to at least write one post per month.

Labels: java

# posted by nileshbansal @ 8:48 pm 8 comments - -   -bookmark at -del.icio.us   -

I finally switched to Firefox 3 on my work PC. It runs opensuse 10.1, which runs an old version of GTK unsupported by Firefox. I did not want to upgrade SuSE just for Firefox, so decided to upgrade just gtk+. The problem is, gtk depends on thousand other packages. And the worst part is no one has documented how to upgrade them, what environment variables to set, and which packages to download. Here is what I did to install all of them, and it was not that bad.

I installed the following packages (in same order):

• glib-2.18.2.tar.bz2
• pixman-0.12.0.tar.gz
• cairo-1.8.4.tar.gz
• pango-1.20.5.tar.bz2
• atk-1.24.0.tar.gz
• gtk+-2.14.2.tar.bz2

I decided to install everything under /opt/gtknew/ in order to keep my already working copy of gtk intact. Below are the environment variables I had to set before compiling all the packages above. For installation, I used ./configure --prefix=/opt/gtknew/; make; make install for all of the packages.

export PKG_CONFIG_PATH=/opt/gtknew/lib/pkgconfig/:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=/opt/gtknew/lib/:$LD_LIBRARY_PATH
export PATH=/opt/gtknew/bin/:$PATH

Once all gtk libraries were installed, all I had to do was to download firefox 3 and set the correct LD_LIBRARY_PATH.

tar jxvf firefox-3.0.4.tar.bz2
cd firefox
export LD_LIBRARY_PATH=/opt/gtknew/lib/:$LD_LIBRARY_PATH
./firefox

I finally edited the firefox executable script, and added the export command for LD_LIBRARY_PATH on the top of that file. Now Firefox uses the new GTK, while rest of the desktop is using the good old version of gtk.

# posted by nileshbansal @ 4:17 pm 5 comments - -   -bookmark at -del.icio.us   -

Recently while playing with user passwords and encryption schemes I realized standards MD5 or SHA are not really standards. At least not for MySQL and Apache. Both claim to implements MD5 and SHA. The only problem is MD5 hash and SHA of same string is different on MySQL and Apache. Both, despite being components of the legendary LAMP stack make no effort of compatibility.

To prove this lets experiment by encoding the string test. Here is what MySQL outputs when using MD5 or SHA1

mysql> select md5("test");
+----------------------------------+
| md5("test")                      |
+----------------------------------+
| 098f6bcd4621d373cade4e832627b4f6 |
+----------------------------------+

mysql> select sha1("test");
+------------------------------------------+
| sha1("test")                             |
+------------------------------------------+
| a94a8fe5ccb19ba61c4c0873d391e987982fbbd3 |
+------------------------------------------+

But here is what Apache expects

[nilesh@krakow ~]$ htpasswd -nbm test test
test:$apr1$EzKMy...$yHUugfRt.9lc.Jc4Z4KzJ/
[nilesh@krakow ~]$ htpasswd -nbs test test
test:{SHA}qUqP5cyxm6YcTAhz05Hph5gvu9M=

Obviously Apache expects different strings than what MySQL produces. Turns out we need to encode MySQL's output in base64 encoding. Here is the code snippet for encoding in Base64 in Java.

System.out.println(new String(
       new org.apache.commons.codec.binary.Base64()
            .encode("098f6bcd4621d373cade4e832627b4f6"
                    .getBytes())));
System.out.println(new String(
       new org.apache.commons.codec.binary.Base64()
            .encode("a94a8fe5ccb19ba61c4c0873d391e987982fbbd3"
                    .getBytes())));

This outputs MDk4ZjZiY2Q0NjIxZDM3M2NhZGU0ZTgzMjYyN2I0ZjY= and YTk0YThmZTVjY2IxOWJhNjFjNGMwODczZDM5MWU5ODc5ODJmYmJkMw==. Even this does not match the Apache's required strings qUqP5cyxm6YcTAhz05Hph5gvu9M= or qUqP5cyxm6YcTAhz05Hph5gvu9M=. To conclude, MySQL and Apache are not compatible, at least not when it comes to encrypting passwords.

I don't know why both projects chose not to implement exactly same algorithm. Anyway, after a few hours of debugging, I decided not to use MySQL's SHA implementation so that the encryption is now taken care by Java on application level using the code snippet in apache's documentation.

UPDATE: Apache adds a random salt to protect the passwords against dictionary attacks. But the value of the salt used is stored with the encrypted string (it is the one between $apr1$ and the following $ sign, i.e., EzKMy in the above example), so I am now confused how would that protect against any kind of attack?

Labels: apache, encryption, mysql, password, security

# posted by nileshbansal @ 12:53 am 7 comments - -   -bookmark at -del.icio.us   -

Dojo comes with a great set of widgets, but is it often necessary to create custom widgets for application specific needs. In this tutorial, I will walk through an example of such a custom widget. The custom widget resides on my own server, while rest of the Dojo (and dijit) is used from AOL CDN hosted xdomain build. Live demo of the widget is also available. This custom widget is built on Dojo 1.0.2 and works great on Firefox 2, Konqueror 3.5.1, IE 6 and IE 7.

We will create a new widget named blogscope.TestWidget which extends the dijit.TitlePane. Looking at the TitlePane code, one can see the empty function onDownloadEnd which is called when the widget finishes downloading content from an external url. For the custom widget, we will extend the TitlePane by filling in this function with a simple alert() statement.

First, we will use dojo.declare to extend the dijit widget to create the new custom widget. I will omit the details of this part, but you can find examples elsewhere.

dojo.provide("blogscope.TestWidget");
dojo.require("dijit.TitlePane");
dojo.declare(
"blogscope.TestWidget",
[dijit.TitlePane],
{
 onDownloadEnd: function(currentConent){
  // summary:
  //  called when download is finished
  alert('Contents Loaded from '+this.href);
 }
}

If you are not using AOL CDN, then simply put the code above in file TestWidget.js. Place this file in directory blogscope such that the dojo path is /js/dojo/ and this widget file is in /js/blogscope/TestWidget.js. Then you can register the widget as dojo.registerModulePath("blogscope", "../blogscope") (the directory name containing the custom widget is relative to the Dojo directory) and use dojo.require("blogscope.TestWidget") in your code. The name of the widget has to be the same as the name of the file containing the code.

Since we wish the xdomain build of Dojo, we need to do some more work. First, the custom widget source code has to be in file TestWidget.xd.js and not TestWidget.js. For the cross-domain build we need to add some more information as to what this module is providing and what all modules it depends on so that Dojo loading system can make sure that the dependencies are met. The contents of this file are as shown below.

dojo._xdResourceLoaded({
depends: [["provide", "blogscope.TestWidget"],
["require", "dijit.TitlePane"]],
defineResource: function(dojo){
 if(!dojo._hasResource["blogscope.TestWidget"]){ 
  dojo._hasResource["blogscope.TestWidget"] = true;
  dojo.provide("blogscope.TestWidget");
  dojo.require("dijit.TitlePane");
  dojo.declare(
  "blogscope.TestWidget",
  [dijit.TitlePane],
  {
   onDownloadEnd: function(currentConent){
    // summary:
    //  called when download is finished
    alert('Contents Loaded from '+this.href);
   }
  }
 );
 }
}});

Note: Ideally, one should use the dojo build system to generate .xd.js files from .js code. But I don't know how to use the build system, so I created the file manually. The main drawback of manual authoring is that the format of .xd.js files may change with version (it is already different for Dojo version 1.1).

Copy the file TestWidget.xd.js on the server such that it is available at say http://www.blogscope.net/js/dojo/blogscope/TestWidget.xd.js. Modify the djConfig variable to set module path as shown below.

<script type="text/javascript"
  djConfig="isDebug: true, parseOnLoad: true, useXDomain: true, modulePaths: {'blogscope': 'http://www.blogscope.net/js/dojo/blogscope'}"
  src="http://o.aolcdn.com/dojo/1.0.2/dojo/dojo.xd.js"></script>
 <script type='text/javascript'>
 dojo.require("blogscope.TestWidget");
 dojo.require("dojo.parser");
 </script>

Since, the module name blogscope is registered to path http://www.blogscope.net/js/dojo/blogscope, dojo.require() adds the a link to http://www.blogscope.net/js/dojo/blogscope/TestWidget.xd.js which can be seen using Firebug.

The widget is now ready to use using the following HTML.

<div dojoType="blogscope.TestWidget" open="false" title="Click to Expand the Widget" href="ajaxContents.html">
</div>

Also See

• Cross Domain Resource Loading in Dojo 0.4.2+ and 0.9
• Timing Issue when loading from CDN

Labels: ajax, dojo, javascript, programming, widget

# posted by nileshbansal @ 6:48 pm 7 comments - -   -bookmark at -del.icio.us   -

I have been delaying purchasing a webcam for past one and a half years. Reason, I don't have windows. And webcams don't work on Linux. And even if they do, MSN is the only IM service you can use (via kopete) for video chat. Well no longer.

Today I purchased a Logitech QuickCam Communicate STX after installing windows on my home desktop. Surprisingly, it works without any pain on my SuSE 10.3 desktop. All I did was to install the driver rpm by adding the repository http://download.opensuse.org/repositories/drivers:/webcam/openSUSE_10.3 followed by installation of RPMs containing string spca. The installed rpms were, gspcav-kmp-default, gspcav1-kmp-default, and spaca5xx-kmp-default. Reboot after installation of the driver made Linux recognize the camera.

Now to use the webcam for video chat, there were two options: MSN via Kopete or Skype via their recently released beta 2.0 client. I prefer skype.

Labels: linux, opensuse, skype, video, webcam

# posted by nileshbansal @ 11:30 pm 5 comments - -   -bookmark at -del.icio.us   -

BlogScope is now indexing blogs from Livejournal. This means, BlogScope users will now be able to search and analyze over 300 posts created by LJ users every minute. Frank the Goat is happy.

Labels: blogs, blogscope, livejournal

# posted by nileshbansal @ 10:32 pm 3 comments - -   -bookmark at -del.icio.us   -