Mostly Harmless

A long time ago, I wrote how bad x64 Java is. Well the good news is -- no more. With 6u14 just around the corner, sanity will be back. Starting update 14, the 64-bit version with Heap sizes below 32GB will use almost the same amount of memory as the 32-bit JVM. All this because of the use compressed pointers to save memory.

It gets even better, the compressed pointers are already available with the Java SE Performance Release since July 2008 (someone should have told me earlier). All that needs to be done is, use -XX:+UseCompressedOops. So I finally decided to give it a try and compare memory usages.

To compare, I used the following test program with 1.6.0_11i686 (32-bit), 1.6.0_11x64 (64-bit), and 1.6.0_06performace with -XX:+UseCompressedOops (64-bit performance) on my desktop with 8GB memory and Intel quad core processor. The minimum amount of memory required to run the program are reported below.

#File: JavaMemoryProblems.java
import java.util.HashMap;
import java.util.Map;
public class JavaMemoryProblems {
   public static void main(String[] args) {
       Map<String, Integer> map = new HashMap<String, Integer>();
       int max = 1000000;
       for (int i=0; i<max; i++) {
           map.put("key"+i, new Integer(i));
       }
       System.out.println("Finished with a map of size "+map.size());
   }
}

The test program is the same as my previous post, and the decrease in the memory is quite amazing. Time to throw the 32-bit JVM out of the window. I haven't done any tests around increase in CPU time, but I do not expect any penalty in run time based on few of my google searches

Update on 31st May 2009: After using the performance release in production for a few days I realize it is broken. The application is performing weirdly, giving incorrect output and throwing exceptions. I have now switched back to the regular release 6u13 release, and everything is fine again. I am not sure if it is really a Java bug or something else, but I am not taking any chances.

Labels: java, performance

# posted by nileshbansal @ 1:03 am 3 comments - -   -bookmark at -del.icio.us   -

Java is by no means a succinct language. For simple operations, the programmer is expected to punch in a hundred keys; but there are a few little shortcuts that can help (and of course Eclipse is always there to help). One such trick is inline initialization of Map which I stumbled across recently and found to be very interesting.

We are all used to using the following code to initialize a map -

Map<String, String> map = new HashMap<String, String>();
map.put("Harry", "Potter");
map.put("Ron", "Weasley");
map.put("Hermione", "Granger");

The problem with this code is it is four different statements. Using a small static initialization trick, you can make it to be a single statement as follows:

Map<String, String> map = new HashMap<String, String>() {{
   put("Harry", "Potter");
   put("Ron", "Weasley");
   put("Hermione", "Granger");
}};

All we are doing here is sub-classing the HashMap class to an anonymous class, and then using the non-static initialization block to call the put() method three times.

PS - I have not updated this blog in a while, as I have been rather busy lately. From now on I will try to find some time, to at least write one post per month.

Labels: java

# posted by nileshbansal @ 8:48 pm 8 comments - -   -bookmark at -del.icio.us   -

We have a lot of data files that needs to be copied to each of the machine we want to run our code on. Further when these data files are updated, they need to be updated on all the machines. This means that the developer has to spend a lot of time just copying these data files around. I wrote a simple solution to this where latest versions of all the data files are maintained at a central server accessible via ssh. These remote date files are copied to the local machine when required in an on-demand fashion transparently by the Java program (after comparing last modification times of the local and remote file).

For accessing files over SFTP, we are using Apache Commons VFS along with Jsch. These libraries (especially commons VFS) is not well documented. I am therefore posting some code snippets from our code documenting the API

The first code snippet demonstrates the API for copying a file from remote location to the local machine:

/**
 * Copies a remote file to local filesystem.
 */
public static void copyRemoteFile(String host, String user,
    String password, String remotePath, String localPath) throws IOException {
    // we first set strict key checking off
    FileSystemOptions fsOptions = new FileSystemOptions();
    SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
            fsOptions, "no");
    // now we create a new filesystem manager
    DefaultFileSystemManager fsManager = (DefaultFileSystemManager) VFS
            .getManager();
    // the url is of form sftp://user:pass@host/remotepath/
    String uri = "sftp://" + user + ":" + password + "@" + host
            + "/" + remotePath;
    // get file object representing the local file
    FileObject fo = fsManager.resolveFile(uri, fsOptions);
    // open input stream from the remote file
    BufferedInputStream is = new BufferedInputStream(fo.getContent()
            .getInputStream());
    // open output stream to local file
    OutputStream os = new BufferedOutputStream(new FileOutputStream(
            localPath));
    int c;
    // do copying
    while ((c = is.read()) != -1) {
        os.write(c);
    }
    os.close();
    is.close();
    // close the file object
    fo.close();
    // NOTE: if you close the file system manager, you won't be able to 
    // use VFS again in the same VM. If you wish to copy multiple files,
    // make the fsManager static, initialize it once, and close just
    // before exiting the process.
    fsManager.close();
    System.out.println("Finished copying the file");
}

Unfortunately the Commons VFS api does not provide a way to check last modification time of a remote file. I had to write that code using the Jsch API. Below is a code snippet that returns last modification time in seconds:

/**
 * Returns a Sftp session conncted using the Jsch library.
 */
public static Session connectSFTP(final String host, final String user,
        final String pass) throws JSchException {
    JSch jsch = new JSch();
    Session session = jsch.getSession(user, host, 22);
    session.setUserInfo(new UserInfo() {
        public String getPassphrase() {
            return null;
        }
        public String getPassword() {
            return null;
        }
        public boolean promptPassphrase(String string) {
            return false;
        }
        public boolean promptPassword(String string) {
            return false;
        }
        public boolean promptYesNo(String string) {
            return true;
        }
        public void showMessage(String string) {
        }
    });
    session.setPassword(pass);
    session.connect();
    return session;
}

/**
 * Returns last modification time of a remote file in seconds.
 */
public static int getLastModificationTime(String host, String user,
        String password, String remotePath) throws IOException,
        JSchException, SftpException {
    Session session = connectSFTP(host, user, password);
    ChannelSftp chan = (ChannelSftp) session.openChannel("sftp");
    chan.connect();
    SftpATTRS attrs = chan.lstat(remotePath);
    int time = attrs.getMTime();
    chan.disconnect();
    session.disconnect();
    return time;
}

I hope that this code is useful to others. Please leave a comment if you see any error of have a suggestion.

Labels: java, sftp

# posted by nileshbansal @ 11:48 pm 30 comments - -   -bookmark at -del.icio.us   -