Mostly Harmless

Java - memory, x64, performance

2009-05-22T01:03:00.016-04:00

A long time ago, I wrote how bad x64 Java is. Well the good news is -- no more. With 6u14 just around the corner, sanity will be back. Starting update 14, the 64-bit version with Heap sizes below 32GB will use almost the same amount of memory as the 32-bit JVM. All this because of the use compressed pointers to save memory.

It gets even better, the compressed pointers are already available with the Java SE Performance Release since July 2008 (someone should have told me earlier). All that needs to be done is, use -XX:+UseCompressedOops. So I finally decided to give it a try and compare memory usages.

To compare, I used the following test program with 1.6.0_11i686 (32-bit), 1.6.0_11x64 (64-bit), and 1.6.0_06performace with -XX:+UseCompressedOops (64-bit performance) on my desktop with 8GB memory and Intel quad core processor. The minimum amount of memory required to run the program are reported below.

#File: JavaMemoryProblems.java
import java.util.HashMap;
import java.util.Map;
public class JavaMemoryProblems {
   public static void main(String[] args) {
       Map<String, Integer> map = new HashMap<String, Integer>();
       int max = 1000000;
       for (int i=0; i<max; i++) {
           map.put("key"+i, new Integer(i));
       }
       System.out.println("Finished with a map of size "+map.size());
   }
}

	with -XX:+UseSerialGC	with -XX:+UseParallelGC
32-bit	110 MB	110 MB
64-bit	180 MB	220 MB
performance	130 MB	160 MB

The test program is the same as my previous post, and the decrease in the memory is quite amazing. Time to throw the 32-bit JVM out of the window. I haven't done any tests around increase in CPU time, but I do not expect any penalty in run time based on few of my google searches

Update on 31st May 2009: After using the performance release in production for a few days I realize it is broken. The application is performing weirdly, giving incorrect output and throwing exceptions. I have now switched back to the regular release 6u13 release, and everything is fine again. I am not sure if it is really a Java bug or something else, but I am not taking any chances.

Initializing Java Maps Inline

2009-04-10T20:48:00.011-04:00

Java is by no means a succinct language. For simple operations, the programmer is expected to punch in a hundred keys; but there are a few little shortcuts that can help (and of course Eclipse is always there to help). One such trick is inline initialization of Map which I stumbled across recently and found to be very interesting.

We are all used to using the following code to initialize a map -


Map<String, String> map = new HashMap<String, String>();
map.put("Harry", "Potter");
map.put("Ron", "Weasley");
map.put("Hermione", "Granger");

The problem with this code is it is four different statements. Using a small static initialization trick, you can make it to be a single statement as follows:


Map<String, String> map = new HashMap<String, String>() {{
   put("Harry", "Potter");
   put("Ron", "Weasley");
   put("Hermione", "Granger");
}};

All we are doing here is sub-classing the HashMap class to an anonymous class, and then using the non-static initialization block to call the put() method three times.

PS - I have not updated this blog in a while, as I have been rather busy lately. From now on I will try to find some time, to at least write one post per month.

Ugrading GTK - Firefox 3 on SuSE 10.1

2008-11-17T16:17:00.004-05:00

I finally switched to Firefox 3 on my work PC. It runs opensuse 10.1, which runs an old version of GTK unsupported by Firefox. I did not want to upgrade SuSE just for Firefox, so decided to upgrade just gtk+. The problem is, gtk depends on thousand other packages. And the worst part is no one has documented how to upgrade them, what environment variables to set, and which packages to download. Here is what I did to install all of them, and it was not that bad.

I installed the following packages (in same order):

glib-2.18.2.tar.bz2
pixman-0.12.0.tar.gz
cairo-1.8.4.tar.gz
pango-1.20.5.tar.bz2
atk-1.24.0.tar.gz
gtk+-2.14.2.tar.bz2

I decided to install everything under /opt/gtknew/ in order to keep my already working copy of gtk intact. Below are the environment variables I had to set before compiling all the packages above. For installation, I used ./configure --prefix=/opt/gtknew/; make; make install for all of the packages.

export PKG_CONFIG_PATH=/opt/gtknew/lib/pkgconfig/:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=/opt/gtknew/lib/:$LD_LIBRARY_PATH
export PATH=/opt/gtknew/bin/:$PATH

Once all gtk libraries were installed, all I had to do was to download firefox 3 and set the correct LD_LIBRARY_PATH.

tar jxvf firefox-3.0.4.tar.bz2
cd firefox
export LD_LIBRARY_PATH=/opt/gtknew/lib/:$LD_LIBRARY_PATH
./firefox

I finally edited the firefox executable script, and added the export command for LD_LIBRARY_PATH on the top of that file. Now Firefox uses the new GTK, while rest of the desktop is using the good old version of gtk.

Password Encryption: MySQL, Apache, and Standards

2008-06-16T00:53:00.013-04:00

Recently while playing with user passwords and encryption schemes I realized standards MD5 or SHA are not really standards. At least not for MySQL and Apache. Both claim to implements MD5 and SHA. The only problem is MD5 hash and SHA of same string is different on MySQL and Apache. Both, despite being components of the legendary LAMP stack make no effort of compatibility.

To prove this lets experiment by encoding the string test. Here is what MySQL outputs when using MD5 or SHA1

mysql> select md5("test");
+----------------------------------+
| md5("test")                      |
+----------------------------------+
| 098f6bcd4621d373cade4e832627b4f6 |
+----------------------------------+

mysql> select sha1("test");
+------------------------------------------+
| sha1("test")                             |
+------------------------------------------+
| a94a8fe5ccb19ba61c4c0873d391e987982fbbd3 |
+------------------------------------------+

But here is what Apache expects

[nilesh@krakow ~]$ htpasswd -nbm test test
test:$apr1$EzKMy...$yHUugfRt.9lc.Jc4Z4KzJ/
[nilesh@krakow ~]$ htpasswd -nbs test test
test:{SHA}qUqP5cyxm6YcTAhz05Hph5gvu9M=

Obviously Apache expects different strings than what MySQL produces. Turns out we need to encode MySQL's output in base64 encoding. Here is the code snippet for encoding in Base64 in Java.

System.out.println(new String(
       new org.apache.commons.codec.binary.Base64()
            .encode("098f6bcd4621d373cade4e832627b4f6"
                    .getBytes())));
System.out.println(new String(
       new org.apache.commons.codec.binary.Base64()
            .encode("a94a8fe5ccb19ba61c4c0873d391e987982fbbd3"
                    .getBytes())));

This outputs MDk4ZjZiY2Q0NjIxZDM3M2NhZGU0ZTgzMjYyN2I0ZjY= and YTk0YThmZTVjY2IxOWJhNjFjNGMwODczZDM5MWU5ODc5ODJmYmJkMw==. Even this does not match the Apache's required strings qUqP5cyxm6YcTAhz05Hph5gvu9M= or qUqP5cyxm6YcTAhz05Hph5gvu9M=. To conclude, MySQL and Apache are not compatible, at least not when it comes to encrypting passwords.

I don't know why both projects chose not to implement exactly same algorithm. Anyway, after a few hours of debugging, I decided not to use MySQL's SHA implementation so that the encryption is now taken care by Java on application level using the code snippet in apache's documentation.

UPDATE: Apache adds a random salt to protect the passwords against dictionary attacks. But the value of the salt used is stored with the encrypted string (it is the one between $apr1$ and the following $ sign, i.e., EzKMy in the above example), so I am now confused how would that protect against any kind of attack?

Creating Custom Dojo Widget using Cross-Domain Build

2008-03-11T18:48:00.021-04:00

Dojo comes with a great set of widgets, but is it often necessary to create custom widgets for application specific needs. In this tutorial, I will walk through an example of such a custom widget. The custom widget resides on my own server, while rest of the Dojo (and dijit) is used from AOL CDN hosted xdomain build. Live demo of the widget is also available. This custom widget is built on Dojo 1.0.2 and works great on Firefox 2, Konqueror 3.5.1, IE 6 and IE 7.

We will create a new widget named blogscope.TestWidget which extends the dijit.TitlePane. Looking at the TitlePane code, one can see the empty function onDownloadEnd which is called when the widget finishes downloading content from an external url. For the custom widget, we will extend the TitlePane by filling in this function with a simple alert() statement.

First, we will use dojo.declare to extend the dijit widget to create the new custom widget. I will omit the details of this part, but you can find examples elsewhere.


dojo.provide("blogscope.TestWidget");
dojo.require("dijit.TitlePane");
dojo.declare(
"blogscope.TestWidget",
[dijit.TitlePane],
{
 onDownloadEnd: function(currentConent){
  // summary:
  //  called when download is finished
  alert('Contents Loaded from '+this.href);
 }
}

If you are not using AOL CDN, then simply put the code above in file TestWidget.js. Place this file in directory blogscope such that the dojo path is /js/dojo/ and this widget file is in /js/blogscope/TestWidget.js. Then you can register the widget as dojo.registerModulePath("blogscope", "../blogscope") (the directory name containing the custom widget is relative to the Dojo directory) and use dojo.require("blogscope.TestWidget") in your code. The name of the widget has to be the same as the name of the file containing the code.

Since we wish the xdomain build of Dojo, we need to do some more work. First, the custom widget source code has to be in file TestWidget.xd.js and not TestWidget.js. For the cross-domain build we need to add some more information as to what this module is providing and what all modules it depends on so that Dojo loading system can make sure that the dependencies are met. The contents of this file are as shown below.


dojo._xdResourceLoaded({
depends: [["provide", "blogscope.TestWidget"],
["require", "dijit.TitlePane"]],
defineResource: function(dojo){
 if(!dojo._hasResource["blogscope.TestWidget"]){ 
  dojo._hasResource["blogscope.TestWidget"] = true;
  dojo.provide("blogscope.TestWidget");
  dojo.require("dijit.TitlePane");
  dojo.declare(
  "blogscope.TestWidget",
  [dijit.TitlePane],
  {
   onDownloadEnd: function(currentConent){
    // summary:
    //  called when download is finished
    alert('Contents Loaded from '+this.href);
   }
  }
 );
 }
}});

Note: Ideally, one should use the dojo build system to generate .xd.js files from .js code. But I don't know how to use the build system, so I created the file manually. The main drawback of manual authoring is that the format of .xd.js files may change with version (it is already different for Dojo version 1.1).

Copy the file TestWidget.xd.js on the server such that it is available at say http://www.blogscope.net/js/dojo/blogscope/TestWidget.xd.js. Modify the djConfig variable to set module path as shown below.


<script type="text/javascript"
  djConfig="isDebug: true, parseOnLoad: true, useXDomain: true, modulePaths: {'blogscope': 'http://www.blogscope.net/js/dojo/blogscope'}"
  src="http://o.aolcdn.com/dojo/1.0.2/dojo/dojo.xd.js"></script>
 <script type='text/javascript'>
 dojo.require("blogscope.TestWidget");
 dojo.require("dojo.parser");
 </script>

Since, the module name blogscope is registered to path http://www.blogscope.net/js/dojo/blogscope, dojo.require() adds the a link to http://www.blogscope.net/js/dojo/blogscope/TestWidget.xd.js which can be seen using Firebug.

The widget is now ready to use using the following HTML.


<div dojoType="blogscope.TestWidget" open="false" title="Click to Expand the Widget" href="ajaxContents.html">
</div>

Also See

Webcam, Video Chat, and Linux

2008-02-05T23:30:00.000-05:00

I have been delaying purchasing a webcam for past one and a half years. Reason, I don't have windows. And webcams don't work on Linux. And even if they do, MSN is the only IM service you can use (via kopete) for video chat. Well no longer.

Today I purchased a Logitech QuickCam Communicate STX after installing windows on my home desktop. Surprisingly, it works without any pain on my SuSE 10.3 desktop. All I did was to install the driver rpm by adding the repository http://download.opensuse.org/repositories/drivers:/webcam/openSUSE_10.3 followed by installation of RPMs containing string spca. The installed rpms were, gspcav-kmp-default, gspcav1-kmp-default, and spaca5xx-kmp-default. Reboot after installation of the driver made Linux recognize the camera.

Now to use the webcam for video chat, there were two options: MSN via Kopete or Skype via their recently released beta 2.0 client. I prefer skype.

Welcome Livejournal

2007-12-13T22:32:00.000-05:00

BlogScope is now indexing blogs from Livejournal. This means, BlogScope users will now be able to search and analyze over 300 posts created by LJ users every minute. Frank the Goat is happy.

Latex to OpenOffice/Word

2007-12-08T13:05:00.000-05:00

Recently I had to convert a big Latex document full of maths to Office format. Copy-paste with all the formulas is not an option (more so, given the terrible state of equation editing features in Microsoft Office 2007). After a lot of searching around, I was able to find ways to automate a large chunk of the process by using LyX and Open Office.

Import Latex document in Lyx.
Export the document in Lyx to HTML. All the math formulas will be converted to images.
Open the HTML document in Firefox.
Start OpenOffice writer, and copy paste all HTML in a new document. Save the document (I used the doc format as I wanted MS Word compatibility). The office document should look similar to the Latex one with all the math equations and symbols as images. The images are referenced as hyperlinks, which means the images will no longer be displayed if you remove the HTML files or copy the file to some other machine. The images should therefore be embedded in the office document.
To embed images in OpenOffice writer, open Edit Menu->Links. Select all images and click on Break Links. This will remove reference to all externally linked images.
The document is now ready. Some manual editing will be required as
- Tables and footnotes were missing
- Images were missing (may be because I was using eps files)
- References to sections, figures, and equations were not always present
- Citations were hyperlinks to some random document

I was able to convert a 10 page latex document to a word document in an hour. The formatting was not very good, as all math equations were slightly vertically displaced and were underlined. But at least it was readable.

Just for reference, I am using Lyx version 1.5.2 with Open Office 2.3 on openSUSE 10.3.

Enhanced AJAXified Tooltips (using Dojo 0.9)

2007-09-15T20:49:00.001-04:00

Tooltips can be very useful widgets for displaying some information quickly. Dojo 0.3 had a nice tooltip widget, which I was able to extend further to use in BlogScope. But in Dojo 0.9, the tooltip widget was re-written with lot of functionality removed. I have managed to create a custom tooltip widget that can be used for displaying large amounts text (that can include images and videos too) and that can fetch the contents by performing an asynchronous HTTP request.

A demo for the new widget is available. The new javascript and CSS files can be downloaded. Explanation for how to use the code and how it works is below.

The file application.js defines two new widgets, blogscope.MasterTooltip and blogscope.Tooltip based on dijit.Tooltip and dijit._MasterTooltip. As an end user you only need to create a blogscope.Tooltip object. Each Tooltip contains a MatserTooltip which stores the actual contents. When the tooltip is opened for the first time, an asynchronous I/O request is dispatched to fetch the contents, and in the meantime a loading icon is displayed. When the contents of the tooltip are displayed, the height and width is clipped to a maximum, which makes the tooltip suitable for displaying large amounts of contents. The tooltip is created right next to the mouse cursor, and the user can move/select/copy from the tooltip. When the mouse moves out of the tooltip area, the MasterTooltip is hidden.

To create the tooltip, following HTML code is required


<span style="color: red" id="tooltipConnection1"> your mouse here (tooltip1)</span>
<div id="tooltipContent1" style="display: none;"></div>

This HTML creates a connection target with id tooltipConnection1, mouse over which will show the tooltip. The DIV tooltipContent1, although never really visible, is required as a placeholder for the actual widget. Following Javascript creates the actual tooltip


blogscopeAddOnLoad( function() {
   var tooltip1 = new blogscope.Tooltip({connectId: "tooltipConnection1", 
       href: "./tooltip.php"}, dojo.byId("tooltipContent1"));
    }
    );

Above code is creating a new object of type blogscope.Tooltip. Note that the javascript is wrapped in blogscopeAddOnLoad, which is a function similar to dojo.addOnLoad, to ensure that the tooltip widget declaration is loaded in advance. When the tooltip widget is created, the node with id tooltipContent1 is replaced by the widget. Two parameters, href and connectId are required for every tooltip. Additional parameters that can be specified include maxWidth, maxHeight and label.

Those who simply wish to use the new Tooltip widget can copy the js and css file, and include it in the HTML (along with dojo 0.9). For those interested in details, the javascript code for the two new widgets is as follows (see application.js for complete code). Note that we want this code to be executed after the Dojo libs finish loading, and therefore everything is inside dojo.addOnLoad.


dojo.addOnLoad( function() {
    dojo.provide("blogscope.Tooltip");
    dojo.provide("blogscope.MasterTooltip");
    // A lot of code here was originaly copied from Tooltip.js in dojo source code, and then modified.
    // Refer to dijit/Tooltip.js for more documentation and explanation
    dojo.declare(
        // Each tooltip has a master tooltip which contains the actual content
        "blogscope.MasterTooltip",
        dijit._MasterTooltip,
        {
            // the fade in fade out duration
            duration: 50,
            parentTooltip: null,
            templateString: "<div id=\"dijitTooltip\" class=\"dijitTooltip dijitTooltipBelow\">\n\t<div class=\"dijitTooltipContainer dijitTooltipContents blogscopeTooltipContainer\" id=\"dijitTooltipContainer\" dojoAttachPoint=\"containerNode\" waiRole='alert'></div>\n\t<div class=\"dijitTooltipConnector blogscopeTooltipConnector\"></div>\n</div>\n",
            show: function(/*DomNode*/ aroundNode){
                if(this.fadeOut.status() == "playing"){
                    // previous tooltip is being hidden; 
                    // wait until the hide completes then show new one
                    this._onDeck=arguments;
                    return;
                }
                this.containerNode.innerHTML=this.parentTooltip.label;
                // Firefox bug. when innerHTML changes to be shorter than previous
                // one, the node size will not be updated until it moves.
                this.domNode.style.top = (this.domNode.offsetTop + 1) + "px";
    
                var align = {'BL' : 'TL'};
                var pos = dijit.placeOnScreen(this.domNode, {x: this.parentTooltip.mouseX, y: this.parentTooltip.mouseY}, ["TL"]);
                this.domNode.className="dijitTooltip dijitTooltipBelow";
    
                // show it
                dojo.style(this.domNode, "opacity", 0);
                this.fadeIn.play();
                this.isShowingNow = true;
                // In our template string, we have 3 div nodes nested in one another
                // for asthetic reasons, we chose to adjust dimension of the innermost DIV
                var adjustNode = getFirstDivChild(getFirstDivChild(this.domNode));
                adjustDimensions(adjustNode, this.parentTooltip.maxWidth, this.parentTooltip.maxHeight);
            },
            // refresh the contents of the tooltip
            refresh: function() {
                this.containerNode.innerHTML=this.parentTooltip.label;
                if (this.isShowingNow == true) {
                    var adjustNode = getFirstDivChild(getFirstDivChild(this.domNode));
                    adjustDimensions(adjustNode, this.parentTooltip.maxWidth, this.parentTooltip.maxHeight);
                }
            },
            // called once after creation of the widget
            postCreate: function(){
                dojo.body().appendChild(this.domNode);
                this.bgIframe = new dijit.BackgroundIframe(this.domNode);
                // I wanted to set a unique id for the domNode, but getUniqueId does not work with IE6
                // If the line below is uncommented, it prints error in IE6
                // this.domNode.id = dijit.getUniqueId(this.declaredClass.replace(/\./g,"_"));
                // Setup fade-in and fade-out functions.
                this.fadeIn = dojo.fadeIn({ node: this.domNode, duration: this.duration, onEnd: dojo.hitch(this, "_onShow") });
                this.fadeOut = dojo.fadeOut({ node: this.domNode, duration: this.duration, onEnd: dojo.hitch(this, "_onHide") });
                // connect the event of mouse moving out
                this.connect(this.domNode, "onmouseout", "_onMouseOut");
            },
            // when mouse moves out
            _onMouseOut: function(/*Event*/ e){
                this.parentTooltip._onMouseOut(e);
            }
        }
    );
    // blogscope.Tooltip is the main widget that we will use
    dojo.declare(
        "blogscope.Tooltip",
        dijit.Tooltip,
        {
            // false if the contents are yet to be loaded from the HTTP request
            hasLoaded: false,
            // location from where to fetch the contents
            href: "",
            // max height and width of the tooltip
            maxWidth: 400,
            maxHeight: 100,
            // the position of mouse when the tooltip is created
            mouseX: 0,
            mouseY: 0,
            // contents to diplay in the tooltip. Initialized to a loading icon.
            label: "<div><img src=\"loading.gif\"> Loading...</div>",
            masterTooltip: null,
            loadContent: function() { 
                if (this.hasLoaded == false) {
                    this.hasLoaded = true;
                    dojo.xhrGet({
                        url: this.href,
                        parentTooltip: this,
                        load: function(response, ioArgs){
                            console.log("data received from ", this.url);
                            console.log("parent is "+this.parentTooltip);
                            this.parentTooltip.label = response;
                            if(this.parentTooltip.isShowingNow){
                                this.parentTooltip.masterTooltip.refresh();
                            }
                        },
                        handleAs: "text"
                    }); 
                }
            },
            open: function(){
                if (this.masterTooltip == null) {
                    // initialized the master tooltip
                    this.masterTooltip = new blogscope.MasterTooltip({parentTooltip: this});
                }
                this.loadContent();
                if(this.isShowingNow){ return; }
                if(this._showTimer){
                    clearTimeout(this._showTimer);
                    delete this._showTimer;
                }
                this.masterTooltip.show(this._connectNode);
                this.isShowingNow = true;
            },
            close: function(){
                if(!this.isShowingNow){ return; }
                if (this.masterTooltip != null) {
                    this.masterTooltip.hide();
                }
                this.isShowingNow = false;
            },
            _onMouseOut: function(/*Event*/ e){
                // currXD and Y are current position of the mouse
                var currX = e.pageX;
                var currY = e.pageY;
                // this.mouseX and this.mouseY are the top-left corners of the tooltip
                var posOffset = Math.abs(this.mouseX - e.pageX) + Math.abs(this.mouseY - e.pageY);
                // console.log("Mouse out called with ",e);
                // we allow mouse movement of 6px
                if (posOffset < 6) {
                    return;
                }
                if(dojo.isDescendant(e.relatedTarget, this._connectNode)){
                     // false event; just moved from target to target child; ignore.
                    return;
                }
                if (this.masterTooltip != null) {
                    // get coordinates and dimensions of the actual tooltip contents
                    var c = dojo.coords(this.masterTooltip.domNode);
                    console.log("Tooltip coordinates", c, "Curr", currX, currY, "MouseXY", this.mouseX, this.mouseY);
                    if (this.mouseX < currX && this.mouseX + c.w > currX && this.mouseY < currY && this.mouseY + c.h > currY) {
                        // the mouse is still on the tooltip contents, no need to close it
                        return;
                    }
                }
                // close the tooltip
                this._onUnHover(e);
            },
            _onHover: function(/*Event*/ e){
                if(this._hover){ return; }
                this._hover=true;
                // find the current mouse postion, the top-left corner of the tooltip will be here
                this.mouseX = e.pageX;
                this.mouseY = e.pageY;
                console.log("Mouse X,Y is "+this.mouseX+", "+this.mouseY);
                // If tooltip not showing yet then set a timer to show it shortly
                if(!this.isShowingNow && !this._showTimer){
                    this._showTimer = setTimeout(dojo.hitch(this, "open"), this.showDelay);
                }
            },
            _onMouseOver: function(/*Event*/ e){
                this._onHover(e);
            }
        }
    );
    // we have finished declaration of the widget
    // now exectute all functions that were waiting for the custom widget to load
    blogscopeTooltipLoaded = true;
    dojo.forEach(blogscopeLoaders,
        function(f) {
            f();
        }
    )
}
);

Code for blogscopeAddOnLoad, which ensures that the declaration of the new widget is loaded before executing more javascript is


// This is a list of functions that need to be executed once the custom widget
// blogscope.Tooltip loads
var blogscopeLoaders = [];
// This variable is set to true after blogscope.Tooltip declaration finishes loading
var blogscopeTooltipLoaded = false;
// This function is similar to dojo.addOnLoad, but it ensures that the
// custom blogscope.Tooltip widget declaration is loaded before the function call is made
blogscopeAddOnLoad = function (/*Function*/ f) {
    if (blogscopeTooltipLoaded) {
        console.log("Calling the function", f);
        f();
    } else {
        console.log("Pushing the function to queue", f);
        blogscopeLoaders.push(f);
    }
}

The code to clip the tooltip to a max width and height is


// Adjusts the dimension of the supplied node so that it does not exceeds
// the max width and height
function adjustDimensions(/*DomNode*/node, /*int*/maxWidth, /*int*/maxHeight) {
    var sHeight = node.scrollHeight;
    var sWidth = node.scrollWidth;
    console.log("height and width are "+sHeight+" and "+sWidth+" for "+node+" with id "+node.id);
    // console.log("max height and width are "+maxHeight+" and "+maxWidth);
    if (sHeight > maxHeight) {
            console.log("resizing to max height "+maxHeight);
            node.style.height = maxHeight + "px";
            node.style.overflow = "auto";
    }
    if (sWidth > maxWidth) {
            console.log("resizing to max width "+maxWidth);
            node.style.width = maxWidth + "px";
            node.style.overflow = "auto";
    }
}

Enjoy!

Hindi Transliteration

2007-08-22T15:10:00.000-04:00

Google has developed a hindi transliteration tool. While this functionality was available for blogger for some time, they have now released it as an independent application that can be used as an iGoogle gadget. This tool converts text typed using roman characters to unicode hindi characters. An ITRANS like scheme is used. Use of roman characters for typing devanagiri is often criticized by the purists, but it for sure is convenient.

The best part is that its written in Javascript. This means that it runs in the browser, and no you don't need Java applets. Also, it is very easy to steal this code for use with your own application (I know the code is unreadable because of the compression).

Shell Command History Size

2007-08-07T18:42:00.000-04:00

A small tip for those who use the command line a lot - add the following to your ~/.bashrc:

export HISTSIZE=9999
export HISTFILESIZE=999999

The value of the HISTSIZE variable determines the number of events preserved in the history list during a session. When you exit from the shell, the most recently executed commands are saved in the file given by the HISTFILE variable (the default is ~/.bash_history). The next time you start the shell, this file initializes the history list. The value of the HISTFILESIZE variable determines the number of lines of history saved in HISTFILE (not necessarily the same as HISTSIZE). HISTSIZE holds the number of events remembered during a session, HISTFILESIZE holds the number remembered between sessions, and the file designated by HISTFILE holds the history list.

Even modern linux distributions set the values for these commands to a low value like 1000. In current world, storing a few 100 kB of history file on disk is nothing. All linux distributions should therefore change the defaults. Till then, add the above mentioned code to your ~/.bashrc and never type the same command twice (instead search using Ctrl + R).

MySQL Lock Contention

2007-07-30T03:21:00.000-04:00

Lock contention can be a serious performance issue with MySQL if both read and write happen concurrently on the same table. BlogScope has tables with around 100 million rows, stored in the MyISAM storage engine. Usual workload consists of 2-3 updates (mainly INSERT) per second and several SELECT operations. Since MyISAM provides only table-level locking, every operation locks the complete table. Although SELECT operations can take place in parallel, it turns out that a long running SELECT can block the complete application. After investigating a bit in why a SELECT was blocking other SELECT operations, I found an explanation in MySQL docs.

A client issues a SELECT that takes a long time to run. Another client then issues an UPDATE on the same table. This client waits until the SELECT is finished.Another client issues another SELECT statement on the same table. Because UPDATE has higher priority than SELECT, this SELECT waits for the UPDATE to finish, and for the first SELECT to finish.

This basically means that if there is any long running SELECT query (e.g., sequential scan), the whole application will just PAUSE for hours. This also means that you can not take backups (using mysqldump), as not only the process willing to write, but even those just reading will be blocked. Fortunately, a fix is available: set the priority of SELECT higher than UPDATE. In this case, the second SELECT statement in the preceding scenario would execute before the UPDATE statement, and would not need to wait for the first SELECT to finish. To set low priority for UPDATES, add the following to /etc/my.cnf

low_priority_updates=1

and execute the following

set global LOW_PRIORITY_UPDATES=1;

While setting updates to be of low priority fixes the locking issue, I don't really like it. In this case, the UPDATE may never actually get a chance to execute if the load is too high. In my opinion, the best solution is to have a replica server. A master MySQL server where all the updates take place, and a slave for all the reads. Once we install our new hardware, I will setup a replica server for BlogScope as well.

Accessing files over SFTP in Java

2007-07-23T23:48:00.000-04:00

We have a lot of data files that needs to be copied to each of the machine we want to run our code on. Further when these data files are updated, they need to be updated on all the machines. This means that the developer has to spend a lot of time just copying these data files around. I wrote a simple solution to this where latest versions of all the data files are maintained at a central server accessible via ssh. These remote date files are copied to the local machine when required in an on-demand fashion transparently by the Java program (after comparing last modification times of the local and remote file).

For accessing files over SFTP, we are using Apache Commons VFS along with Jsch. These libraries (especially commons VFS) is not well documented. I am therefore posting some code snippets from our code documenting the API

The first code snippet demonstrates the API for copying a file from remote location to the local machine:


/**
 * Copies a remote file to local filesystem.
 */
public static void copyRemoteFile(String host, String user,
    String password, String remotePath, String localPath) throws IOException {
    // we first set strict key checking off
    FileSystemOptions fsOptions = new FileSystemOptions();
    SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
            fsOptions, "no");
    // now we create a new filesystem manager
    DefaultFileSystemManager fsManager = (DefaultFileSystemManager) VFS
            .getManager();
    // the url is of form sftp://user:pass@host/remotepath/
    String uri = "sftp://" + user + ":" + password + "@" + host
            + "/" + remotePath;
    // get file object representing the local file
    FileObject fo = fsManager.resolveFile(uri, fsOptions);
    // open input stream from the remote file
    BufferedInputStream is = new BufferedInputStream(fo.getContent()
            .getInputStream());
    // open output stream to local file
    OutputStream os = new BufferedOutputStream(new FileOutputStream(
            localPath));
    int c;
    // do copying
    while ((c = is.read()) != -1) {
        os.write(c);
    }
    os.close();
    is.close();
    // close the file object
    fo.close();
    // NOTE: if you close the file system manager, you won't be able to 
    // use VFS again in the same VM. If you wish to copy multiple files,
    // make the fsManager static, initialize it once, and close just
    // before exiting the process.
    fsManager.close();
    System.out.println("Finished copying the file");
}

Unfortunately the Commons VFS api does not provide a way to check last modification time of a remote file. I had to write that code using the Jsch API. Below is a code snippet that returns last modification time in seconds:


/**
 * Returns a Sftp session conncted using the Jsch library.
 */
public static Session connectSFTP(final String host, final String user,
        final String pass) throws JSchException {
    JSch jsch = new JSch();
    Session session = jsch.getSession(user, host, 22);
    session.setUserInfo(new UserInfo() {
        public String getPassphrase() {
            return null;
        }
        public String getPassword() {
            return null;
        }
        public boolean promptPassphrase(String string) {
            return false;
        }
        public boolean promptPassword(String string) {
            return false;
        }
        public boolean promptYesNo(String string) {
            return true;
        }
        public void showMessage(String string) {
        }
    });
    session.setPassword(pass);
    session.connect();
    return session;
}

/**
 * Returns last modification time of a remote file in seconds.
 */
public static int getLastModificationTime(String host, String user,
        String password, String remotePath) throws IOException,
        JSchException, SftpException {
    Session session = connectSFTP(host, user, password);
    ChannelSftp chan = (ChannelSftp) session.openChannel("sftp");
    chan.connect();
    SftpATTRS attrs = chan.lstat(remotePath);
    int time = attrs.getMTime();
    chan.disconnect();
    session.disconnect();
    return time;
}

I hope that this code is useful to others. Please leave a comment if you see any error of have a suggestion.

Finally uninstalled ZMD

2007-06-02T13:07:00.000-04:00

ZMD is the worst thing that happened to SuSE. It often starts eating 100% of CPU for no reason, crashes every now and then, and returns "unresolvable id" error most of the time when trying to install updates. Good that they will remove ZMD from 10.3 completely. ZMD can be uninstalled from 10.2 as well (although it is installed by default). For some reason I was reluctant to uninstall it from my laptop, despite it (and some md5sum, gzip and parse-metadata processes) eating 100% CPU all the time. But yesterday, I finally uninstalled it and it feels so much better.

Uninstallation was simpler than I thought. Just two commands

rczmd stop
rpm -e zmd libzypp-zmd-backend sqlite-zmd rug zen-updater

After removing ZMD, selecting Install Software starts YaST. To check for updates, I started openSUSE Update Applet from System -> Desktop Applets. This applet is not as beautiful as the previous ZMD one, but it works better (no "unresolvable id" errors).

Those interested in configuring SuSE for package management should read the following openSUSE community article.

Wireless Disabled After Resume from Suspend

2007-04-24T00:12:00.000-04:00

After I resume my laptop (OpenSUSE 10.2, Dell 640m, Intel 3945ABG wireless card) from stand by (suspend to ram), the wireless usually stops working. The KNetworkManager applet stops showing the wireless device. After investigating further, it seems that the problem is with the HAL daemon, and restarting it fixes the problem.

 /etc/init.d/haldaemon restart

However restarting HAL daemon manually every time after resume was not a good option. So I decided to use my time while waiting at the SF airport better by finding a solution to the problem. It seems that SuSE uses pm-utils for managing suspend and other power related options stating version 10.2. Some of the configurations are present in /etc/pm/config. It turns out that all the scripts present in /etc/pm/hooks/ are executed after every suspend or resume event. One of the following four option is passed as a command line argument to each of the script: hibernate, suspend, thaw, or resume. Since I wanted to restart haldaemon at every resume, all I had to do was to create a new executable file in /etc/pm/hooks/ with the following contents:

#!/bin/sh
# File: /etc/pm/hooks/99haldaemon

case "$1" in
        resume)
                /etc/init.d/haldaemon restart
                ;;
        *)
                ;;
esac

And now the haldaemon is restarted at every resume, and the wireless works fine after I open the laptop lid. This is a hack, and I hope a proper fix will be included in future versions of SuSE.

Lucene Optimization Tip: Reuse Searcher

2007-04-11T15:42:00.000-04:00

Lucene is a great text search engine, but some aspects of it are not clearly documented. For example, if you have multiple simultaneous requests (e.g., in a web application) searching for different queries, should there be a separate IndexSearcher for each thread or one shared IndexSearcher will do.

Initially we were using different IndexSearcher for different requests, creating a new instance at the start of the request and destroying soon after searching. After a lot of experimentation and exchange of a few mails over the lucene mailing list, I discovered that the efficient way is to use a single shared IndexSearcher across all requests. Multiple concurrent threads can easily invoke the search method on a single IndexSearcher object. Reusing cached data makes the single searcher approach very efficient both in terms of response time and memory usages. In our case, response times decreased by a factor of 2-3 after this change.

Remember that IndexSearcher object however needs to be destroyed and recreated after the index is modified or updated. Unless the searcher is recreated, it will not reflect the changes made to the index. A good strategy is to periodically destroy and recreate the shared index using a separate Timer thread.

Prevent Timeouts in SSH

2007-02-23T18:55:00.000-05:00

Many routers timeout the connection after some period of inactivity. For users of ssh, this can a big nuisance. In presence of such routers a ssh connection with no activity (for a few minutes) will disconnect automatically. The workaround is that the server and client keep talking to each other even when the user is not typing (or viewing) anything. This can be achived using one of the two ways:

Server Side

Add the following two lines to /etc/ssh/sshd_config. This will make the server poll for client every 30 seconds. If the client fails to respond for 4 times (i.e., 2 mins), the server will close the connection. Note that this applies to version 2 of OpenSSH only.

ClientAliveInterval 30
ClientAliveCountMax 4

Client Side

Add the following two lines to /etc/ssh/ssh_config (or $HOME/.ssh/config if you don't have root access). his will make the client poll for server every 30 seconds. If the servers fails to respond for 4 times (i.e., 2 mins), the client will close the connection. Note that this also applies to version 2 of OpenSSH only.

ServerAliveInterval 30
ServerAliveCountMax 4

Simple Software RAID in Linux

2007-02-23T15:41:00.000-05:00

RAID is commonly used in production environments to spread the data among multiple disks for higher performance, reliability or capacity. This is a small tutorial for quick setup of RAID 0 or 1 under Linux using mdadm. Fore more advanced configuration and options look elsewhere. Just for records, I have tested this on RHEL 4 update 4.

This document outlines creation of a RAID 0 drive consisting of three partitions /dev/sdb1, /dev/sdc1, and /dev/sdd1. Procedure for RAID 1 should be similar. First we need to prepare the three partitions for RAID. Use fdisk and set partition's system id of all three partitions to fd (Linux raid auto). You may need to reboot after this.

[root@trinity]# fdisk /dev/sdb1
<press t to change partition id>
<type fd as partition id>
<press w to write changes to disk>

Use mdadm to create the array.

mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --chunk=512

This will create a new drive /dev/md0 with chunk size 512KB. Choice of chunk size depends on application. Default chunk size is 64K. This new drive now ready for use. There is no need to sotre these configurations in any file as mdadm remembers everything for you even after reboot. For example, to create a new ext3 filesystem here use mkfs.ext3 /dev/md0.

Information about the active RAID arrays is displayed in /proc/mdstat. mdadm --detail --scan can also be used.

[root@matrix]# cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 sdd1[2] sdc1[1] sdb1[0]
      215043072 blocks 1024k chunks

unused devices:

SuSE, Acer aspire 5580 and non-standard resolution 1280x800

2007-01-28T23:50:00.000-05:00

My roommate recently purchased an Acer Aspire 5580 laptop and handed it to me for installing Linux. It is a pretty good laptop, well supported under Linux. It features Intel Core 2 Duo processor with Intel 945 GMA and 3945ABG wireless. I installed openSUSE 10.2 and everything went fine except for the screen resolution.

This laptop uses 1280x800 as native resolution, but my X refused to work with anything greater than 1024x768. Initially I thought that the problem is with SuSE not recognizing HSync and VSync frequencies of the LCD screen. I tried every possible refresh rate in display settings. After a lot of googling around, I found 945gm and 1280x800 laptop monitor problem suse 10.2 and patch the video BIOS. For some reason, the hardware vendor did not include the native resolution of the LCD display in the list of supported resolutions. The solution however is simple, as there exists a hack for fooling X. On SuSE, simply edit /etc/sysconfig/videobios and reboot. Also make sure that /etc/init.d/boot.videobios is executed on boot time by running the following command.

insserv boot.videobios

After editing, my /etc/sysconfig/videobios looks as follows:

VIDEOBIOS_PATCH="yes"
VIDEOBIOS_PARAMETERS="5c 1280 800"

I don't understand why hardware vendors can not follow standards! Anyway, the laptop is functioning fine now.

BSNL broadband using Linux

2006-12-05T23:08:00.000-05:00

BSNL DataOne provides broadband services using ADSL. They provided me with a Sterlite modem that can be used for pppoe (since it is configured to operate in pure bridged mode). While only windows is supported officially, it is easy to use this modem with Linux. I am using OpenSUSE 10.1 on my laptop. The details for configuration are as follows:

You first need to install rp-pppoe. RPM for SuSE 10.1 is available. To configure, pppoe-setup can be used, which will ask for many simple questions. I had the modem connected to my lan card (which was configured as eth0). My answers were as follows.

Welcome to the Roaring Penguin PPPoE client setup.  First, I will run
some checks on your system to make sure the PPPoE client is installed
properly...

Looks good!  Now, please enter some information:
USER NAME
>>> Enter your PPPoE user name:dsaXYZXYZ

INTERFACE
>>> Enter the Ethernet interface connected to the DSL modem
For Solaris, this is likely to be something like /dev/hme0.
For Linux, it will be ethn, where 'n' is a number.
(default eth0):eth0

MODEM TYPE
We will try to detect if your modem is compliant with RFC 2516
or not. 3COM's 3CP4130 is *NOT* compliant, for instance.

Searching for a modem at interface eth0...
Found a RFC 2516 compliant modem, congratulations! :)
Do you want the link to come up on demand, or stay up continuously?
If you want it to come up on demand, enter the idle time in seconds
after which the link should be dropped.  If you want the link to
stay up permanently, enter 'no' (two letters, lower-case.)
NOTE: Demand-activated links do not interact well with dynamic IP
addresses.  You may have some problems with demand-activated links.
>>> Enter the demand value (default no):

DNS
Please enter the IP address of your ISP's primary DNS server.
If your ISP claims that 'the server will provide DNS addresses',
enter 'server' (all lower-case) here.
If you just press enter, I will assume you know what you are
doing and not modify your DNS setup.
>>> Enter the DNS information here: server

PASSWORD
>>> Please enter your PPPoE password:
>>> Please re-enter your PPPoE password:

FIREWALLING
Please choose the firewall rules to use.  Note that these rules are
very basic.  You are strongly encouraged to use a more sophisticated
firewall setup; however, these will provide basic security.  If you
are running any servers on your machine, you must choose 'NONE' and
set up firewalling yourself.  Otherwise, the firewall rules will deny
access to all standard servers like Web, e-mail, ftp, etc.  If you
are using SSH, the rules will block outgoing SSH connections which
allocate a privileged source port.

The firewall choices are:
0 - NONE: This script will not set any firewall rules.  You are responsible
          for ensuring the security of your machine.  You are STRONGLY
          recommended to use some kind of firewall rules.
1 - STANDALONE: Appropriate for a basic stand-alone web-surfing workstation
2 - MASQUERADE: Appropriate for a machine acting as an Internet gateway
                for a LAN
>>> Choose a type of firewall (0-2): 0

** Summary of what you entered **
Ethernet Interface: eth0
User name:          dsa220513
Activate-on-demand: No
DNS addresses:      Supplied by ISP's server
Firewalling:        NONE

To start, stop and to view the status, pppoe-start, pppoe-stop and pppoe-status can be used. I expected the internet to work, since after running pppoe-start, status showed that I am connected. Messages in /var/log/messages also didn't contain any error. LAN card had acquired IP address 192.168.1.2 from the DHCP of the modem. Modem's web configuration interface was accessible at http://192.168.1.1/.

pppoe-status: Link is up and running on interface ppp0
ppp0      Link encap:Point-to-Point Protocol
          inet addr:59.95.164.21  P-t-P:59.95.160.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:6727 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5687 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:6424287 (6.1 Mb)  TX bytes:826896 (807.5 Kb)

After some investigation I found that if I set my ethernet card (using YaST) to have static IP 192.168.0.3, and configure its default gateway as 192.168.1.1 everything works fine. I am not sure what is the reason for this, as I have no experience with pppoe; but it works. I will try to dig in this and find out the reason.

My new Dell 640m and Linux

2006-11-04T04:59:00.000-05:00

Few days back I treated myself with purchase of a Dell Inspiron 640m. Its a Core 2 Duo (T5500/2MB Cache/1.66GHz/667MHz FSB) machine with Intel chipset and Intel 3945 internal wireless. I installed openSUSE 10.1, and removed designed for windows sticker to stick it on my trash can. Everything worked fine with out-of-the-box install. Here is a detailed account:

Wireless: works fine in out-of-the-box install. Keyboard key to enable/disable wireless also works. The only way however to find out if the wireless is disabled by checking dmesg, which contains the following message when wireless is disabled.

ipw3945: Radio Frequency Kill Switch is On:
Kill switch must be turned off for wireless networking to work.

Onboard Video Card: Works fine. Intel's open source driver for 945GM support 3D accelration. I was able to play supertux without any problems. Even Xgl worked flawlessly. But, I was not able to play Warcraft III using Cedega as the framerate was too bad. Native version of UT2004 also didn't work. Probably this card is not meant for gaming.

Dual head mode was not very smooth as resolution sometimes messed up after switching the output device. But it seems like a problem with SuSE and YaST, and not with the drivers.

Sound: Works fine most of the time. But sometimes I need to delete and re-add the sound card using yast for audio to work. This is some bug in the driver as my dmesg is full of error messages.

ALSA sound/core/control.c:635: BUG? (info->access == 0)
[] snd_ctl_ioctl+0x463/0x9ea [snd]
[] snd_ctl_ioctl+0x0/0x9ea [snd]
[] do_ioctl+0x1c/0x5e
[] vfs_ioctl+0x24c/0x25e
[] do_fcntl+0x1c9/0x249
[] sys_ioctl+0x51/0x68
[] sysenter_past_esp+0x54/0x79

SD card reader: Doesn't work! But I can use my digital camera to transfer using usb. It seems that Gentoo wiki has instructions to enable this support.

Additional keyboard keys: All useful keys like changing LCD brightness, switching video output etc.., work fine. Other additional keys work after setup in GNOME (but not in KDE).

TV out and Express card: not tested

Overall, I am happy with my choice and its Linux compatibility.

Extending dojo.widget.Tooltip

2006-10-17T00:07:00.000-04:00

I have been playing with Dojo for past few weeks (since we are using it for AJAX stuff at BlogScope). While it is a nice library, many parts are not well documented. One such area is extending widgets to add to their functionality. Tooltip is a good example of widget that requires improvement, since it does not provide an option to have some custom text displayed while the contents of tooltip are being fetched. For BlogScope, I wanted to have an loading icon displayed while the tooltip is being fetched in background. This article explains the code that I am using for this purpose. Note: I am no Dojo guru. This code may not be the best way to do this, but it works.

Demo: We have a demo page displaying the extended tooltip. This demo page also has javascript to resize the tooltip to specified max-height and max-width. While CSS can be used for max-height and max-width, to make it work in IE, javascript is required. View source of the demo page to get the complete javascript code.

Below is the HTML code for the tooltip. There are three main components. One is the link (with id=one), mouseover on which displays the tooltip. Second is the div element with dojoType="PostTooltip", which specifies the url from where contents of tooltip are to be fetched. Third component is the div block (id=two) that is the tooltip itself. After the contents are being fetched, innerHTML of this div block is updated.

<a href="some-page" id="one">bring your mouse here</a>
<div dojoType="PostTooltip" style="display:none;"
    dojo:connectId="one"
    dojo:uniqId="two"
    dojo:contentUrl="tooltip.php?id=two"
    dojo:executeScripts="true"
    >
    <div id="two" style="background-color: white; border: 1px dashed #aaaaaa;">
            this is a custom loading message.
    </div>
</div>

Below is the code for fetching the contents of tooltip and adjusting its dimensions. uniqId is the id of div block for the tooltip. Since the mimetype is text/javascript, the fetched contents must be a javascript source which when executed updates the innerHTML of tooltip's div block (something like document.getElementById(uniqId).innerHTML = 'This is the new content';).

function updatePostTooltip(postUrl, uniqId) {
        dojo.io.bind({
            url: postUrl,
            load: function(type, evalObject){
                dojo.debug("data received from "+postUrl);
                adjustPostTooltipDimensions(uniqId);
            },
            mimetype: "text/javascript"
        });
}

function adjustPostTooltipDimensions(uniqId) {
        var sHeight = document.getElementById(uniqId).scrollHeight;
        var sWidth = document.getElementById(uniqId).scrollWidth;
        var maxHeight = 450;
        var maxWidth = 480;
        dojo.debug("height and width are "+sHeight+" and "+sWidth+" for "+uniqId);
        if (sHeight > maxHeight) {
                dojo.debug("resizing to max height "+maxHeight);
                document.getElementById(uniqId).style.height = maxHeight + "px";
                document.getElementById(uniqId).style.overflow = "auto";
        }
        if (sWidth > maxWidth) {
                dojo.debug("resizing to max width "+maxWidth);
                document.getElementById(uniqId).style.width = maxWidth + "px";
                document.getElementById(uniqId).style.overflow = "auto";
        }
    }

And finally below is the code for extended Tooltip widget.

dojo.provide("my.widget.PostTooltip");
    dojo.widget.manager.registerWidgetPackage("my.widget");

    dojo.widget.defineWidget(
        "my.widget.PostTooltip",
        dojo.widget.html.Tooltip,
        {
                //we have added three new variables, uniqId, contentUrl and hasContents
                uniqId: "",
                contentUrl: "",
                hasContents: 0,
                //We modify this function from the original in dojo.widget.html.Tooltip
                onShow: function() {
                        // copied as is from dojo/src/widget/html/Tooltip.js
                        dojo.widget.html.Tooltip.superclass.onShow.call(this);
                        this.state="displayed";
                        if(this.eraseScheduled){
                                this.hide();
                                this.eraseScheduled=false;
                        }
                        //end of copied code
                        //this is our code
                        if (this.hasContents == 0) {
                                updatePostTooltip(this.contentUrl, this.uniqId);
                                this.hasContents = 1;
                        }
                }
        }
    );

Firefox, Linux and Audio

2006-09-15T05:50:00.000-04:00

Popularity of flash based video distribution sites like YouTube has made Firefox with its flash player an important component of today's desktop. But it is often the case with not-so-new (SuSE 10.0 and earlier) Linux desktops that audio refuses to work. Most common reason for audio not working is when some other process or program has taken control of the sound device, and Flash player can not access it. Fortunately, fixing this problem is easy, provided your system is not too old.

Fix: Firefox is started by a shell script named firefox, usually located in /usr/bin/. This script executes the mozilla firefox binary with appropriate options. To fix the sound problem, you need to start this binary with aoss. This translates to editing /usr/bin/firefox as root, and modifying the line containing

$MOZ_PROGRAM $@

aoss $MOZ_PROGRAM $@

Note that these instructions are tested with SuSE 9.3 and 10.0, and some things may vary depending on the distro in use.

Some history: Old linux used oss as its sound interface. OSS is not capable of sound mixing that enables multiple programs to use the sound device simultaneously. KDE's aRts was used on top of OSS for a long time as a temporary solution as aRts provided software mixing. But aRts is buggy, and is not well supported by many non-KDE applications. Finally, OSS was replaced by a new advanced sound interface, ALSA in 2002. ALSA can use hardware capabilities of audio device to mix sound coming from multiple sources, and in case hardware does not support sound mixing, ALSA provides a software implementation of mixer named dmix.

Flash player: Flash is a proprietary software, owned by a commercial organization with sole purpose of making money. It still uses OSS, as porting it to use ALSA would not have created any extra money for Macromedia/Adobe. aoss is a script that can provide OSS emulation to applications requiring OSS, while actually using ALSA. So when Firefox is run with aoss, Flash player uses ALSA in place of OSS. On newer linux distributions (openSUSE 10.1 and later), Firefox is configured by default to use aoss.

Launching BlogScope

2006-08-27T02:28:00.000-04:00

Internet has provided a platform using which everyone can express his or her opinion to a wider audience. Millions of people from all around the world are using blogs for documenting their life; people write their experiences about everything, from movies to restaurants, and politics to personal life. This phenomenal popularity of blogs as media form has made blogosphere an invaluable source of information. But, it is not an easy task to find relevant and interesting content, as there are millions of blogs and a lot of content is repetitive and noisy. For past seven months, we have been working on how can we assist the user in his or her journey of exploration through blogosphere. As a result, we have developed BlogScope.

BlogScope is a knowledge discovery and analysis tool for blogosphere. To assist user find interesting information from large amount of text is blogosphere, BlogScope analysis engine offers tools like popularity curves, correlated terms, popularity bursts, and comparison curves. These tools can be used to track trends and temporal evolution of topics, investigate the relationship between keywords, or even for comparing apples with oranges. For bloggers, BlogScope offers some interesting tools like Summary Cloud. To know more read the about page. We also have a small 90 second flash movie demonstrating the system.

We are currently tracking over 4 million blogs with 25 million blog posts in our database. Number of posts is growing at a fast rate of hundred thousand per day. Codebase of BlogScope is also growing at an equivalently fast rate, and we plan to add many more cool features in future, so keep checking. We hope that you like it. And do send your feedback.

This is a flash demo of BlogScope and requires availability of flash player.

Our promotional logos:

Last updated: 11th Sept, 2006.

java Exception - Unknown Source

2006-08-08T01:54:00.000-04:00

When debugging my Java programs, reading through the exception stack trace I realized that many a times the line number in was not reported. Just the method name and file name with unknown source was reported. For example consider the following simple program compiled with build file as shown

//File: TestDebug.java
public class TestDebug {
    public static void main(String[] args) {
        ((String)null).length();
    }
}

<!-- build.xml -->
<project name="TestProject" default="build" basedir=".">
 <target name="build">
  <javac srcdir="." destdir="."/>
 </target>
</project>

[nilesh^trinity]$ ant build
...
[nilesh^trinity]$ java TestDebug
Exception in thread "main" java.lang.NullPointerException
        at TestDebug.main(Unknown Source)

After some investigation I found that ant omits debug information from compiled class by default. Hence producing exact line number in case of an error is not possible. The solution however is simple. I just modified my build file to turn debug on and to set debuglevel="lines,vars,source" under javac target in ant, and everything worked fine. Modified build file for above example will be:

<!-- build.xml -->
<project name="TestProject" default="build" basedir=".">
 <target name="build">
  <javac srcdir="." destdir="." debug="on" debuglevel="lines,vars,source" />
 </target>
</project>

Line numbers are now reported with the stack trace:

[nilesh^trinity]$ ant build
...
[nilesh^trinity]$ java TestDebug
Exception in thread "main" java.lang.NullPointerException
        at TestDebug.main(TestDebug.java:4)

AMD64: java.lang.OutOfMemoryError

2006-07-15T19:17:00.000-04:00

Today I discovered that AMD64 version of Sun Java VM 1.5.0 has some memory problems.

I spent last couple of weeks profiling a Java web application (running with Tomcat). The application crashed after running for 2-3 days with java.lang.OutOfMemoryError (even with heap size 4GB). My development machine is 32-bit and production machine 64-bit which confused me more as profiling was not able to find anything suspicious. Finally I tried using i586 JRE on our 64-bit production machines and things worked fine. Probably amd64 garbage collector has some bug. Lesson: Java HotSpot amd64 VM 1.5 has memory management problems! use 32-bit VM instead.

To reproduce this effect, I have following simple program:

#File: JavaMemoryProblems.java
import java.util.HashMap;
import java.util.Map;
public class JavaMemoryProblems {
   public static void main(String[] args) {
       Map<String, Integer> map = new HashMap<String, Integer>();
       int max = 1000000;
       for (int i=0; i<max; i++) {
           map.put("key"+i, new Integer(i));
       }
       System.out.println("Finished with a map of size "+map.size());
   }
}

When I run this program with i586 and amd64 versions of Java VM 1.5.0_07, I get the following results:

32-bit: java -Xmx120m JavaMemoryProblems. With 120MB memory it works fine.
64-bit: java -Xmx170m -XX:+UseSerialGC JavaMemoryProblems. Anything less than 180MB throws OutOfMemory (requires 50% more than i586 version).
64-bit: java -Xmx200m -XX:+UseParallelGC JavaMemoryProblems. With 200MB but with ParallelGC (default on server class machines) hangs and needs to be killed with kill -9. In this case the parallel GC thread keeps calling garbage collector repeatedly. Lesson: Use -XX:+UseSerialGC if running low on memory, ParallelGC takes too much time and CPU (and even hangs) before it figures out it can not continue.

After switching to i586 VM I am saving a lot of memory (30-40%). Earlier a simple search program required 250MB heap size to search a Lucene index with around 15 million docs (10GB index size), now it needs just 150MB. Maintaining statistics consisting of around 100 Tries each with roughly 1 million entries required over 1050MB with amd64 VM, now I can do the same with 650MB. Memory required for parsing a simple 55MB XML from weblogs using JDOM with SAXBuilder with i586 VM is 290MB memory compared to 440MB for amd64. Similiar things are also mentioned by sotua in Ubuntu forums.

Anyway I have learned many interesting things about Java VM and garbage collection in last 2 weeks:

Netbeans Profiler is better than Eclipse TPTP.
Java debugging options -verbose:gc, -XX:+PrintGCDetails and -XX:+PrintGCTimestamps are really useful.
Tuning GC with Java 5.0 is an interesting read.
Difference between PermSize and HeapSize.

Profiling with Eclipse

2006-07-11T03:18:00.000-04:00

This article explains setup required for profiling Java application (including tomcat) with eclipse using TPTP platform. I am using Linux with i386 Sun JDK 1.5.0 (it seems that 64-bit version of agent controller is not available). In order to use TPTP, one must install two things: agent controller and eclipse tptp plugin.

Agent Controller: Agent controller needs to be installed on machine where the java process will run.

1. Download the agent controller and unzip it.

mkdir $HOME/usr/tptp-agent
cd $HOME/usr/tptp-agent/
unzip tptpdc.linux_ia32-TPTP-4.2.0.zip

2. Set up the environment variables by adding the following in your .bashrc after adjusting the JAVA_HOME

export JAVA_HOME=/opt/java/jdk1.5.0_06/
export RASERVER_HOME=$HOME/usr/tptp-agent/
export LD_LIBRARY_PATH=$RASERVER_HOME/lib/:$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/i386/server/
export PATH=$JAVA_HOME/bin/:$PATH:$RASERVER_HOME/bin/

3. Configure the agent controller by running the setup script from bin directory. When asked for network configuration, provide a list of allowed hosts or allow everyone.

cd $HOME/usr/tptp-agent/bin/
./SetConfig.sh

4. Edit RAStart.sh script in bin directory to point to correct RASERVER_HOME

5. Start agent controller using the command RAStart.sh. To stop use RAStop.sh.

TPTP Eclipse Plugin: First check requirements for installing TPTP. Atleast JDK 1.4, Eclipse 3.2 and EMF are required. Update manager can be used to install TPTP using remote URL http://eclipse.org/tptp/updates/site.xml. If you don't have BIRT installed (which is an optional requirement), unselect features needing BIRT. On successful completion of installation, Profiling and Logging will show up in list of available perspectives.

Start Profiling: After starting the agent controller, start the long running Java process to profile with extra argument -XrunpiAgent:server=enabled.

java -XrunpiAgent:server=enabled MyJavaProgram

For profiling Tomcat, add this in catalina.sh under JAVA_OPTS. Start eclipse, select profile from toolbar -> Attach to Java process. Specify the host in the Host tab and select the process to profile in the Agents tab. From the Profiling tab select profiling type and click on Profile to start.

There are a bunch of profiling views available. Refer to eclipse tptp documentation for more help.

SSH brute attack

2006-06-23T02:36:00.000-04:00

Today I discovered that one of our lab machines has been compromised. It faced heavy dictionary attack via ssh from matheron.scv.math.unb.ca. One of the users had a simple password and the attacker was able to get access to his account. Attackers after getting the access, copied a brute force cracking program for ssh to /tmp/. /brute and started using our machine to break in other machines.

I had noticed such brute force attacks earlier, but ignored them. I never realized that such attacks actually work. In last one month, there were over 20 thousand such attempts from over 100 different hosts. Scary!

An excerpt from /var/log/messages/

Jun 18 05:20:17 soho sshd(pam_unix)[29250]: check pass; user unknown
Jun 18 05:20:17 soho sshd(pam_unix)[29250]: authentication failure; logname= uid=0
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173
Jun 18 05:20:21 soho sshd(pam_unix)[29252]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173  user=root
Jun 18 05:20:26 soho sshd(pam_unix)[29254]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173  user=root
Jun 18 05:20:33 soho sshd(pam_unix)[29256]: check pass; user unknown
Jun 18 05:20:33 soho sshd(pam_unix)[29256]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173
Jun 18 05:20:38 soho sshd(pam_unix)[29258]: check pass; user unknown
Jun 18 05:20:38 soho sshd(pam_unix)[29258]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173
Jun 18 05:20:45 soho sshd(pam_unix)[29260]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173  user=root
Jun 18 05:20:50 soho sshd(pam_unix)[29262]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173  user=root
Jun 18 05:20:55 soho sshd(pam_unix)[29264]: check pass; user unknown
Jun 18 05:20:55 soho sshd(pam_unix)[29264]: authentication failure; logname= uid=0 
                                   euid=0 tty=ssh ruser= rhost=202.96.216.173

The reason I was able to detect that our machine is compromised is that brute program crashed frequently and had entries in /var/log/messages/

Jun 20 21:51:33 soho kernel: brute[15536]: segfault at 0000000000000000 
                                                rip 0000000008048e33 rsp 00000000ffffd280 error 4
Jun 20 21:51:33 soho kernel: brute[15537]: segfault at 0000000000000000 
                                                rip 0000000008048e33 rsp 00000000ffffd280 error 4
Jun 20 21:51:33 soho kernel: brute[15538]: segfault at 0000000000000000 
                                                rip 0000000008048e33 rsp 00000000ffffd280 error 4
Jun 20 21:51:33 soho kernel: brute[15540]: segfault at 0000000000000000 
                                                rip 0000000008048e33 rsp 00000000ffffd280 error 4

The bottom line is: Its a dangerous world out there!

SuSE 10.1 Installation Guide

2006-05-22T18:04:00.000-04:00

I recently installed SUSE Linux 10.1 on my workstation. While installing, I saved screenshot for each step and prepared a step by step guide to installing SUSE 10.1 for newbies.

VMWare Workstation on SuSE Linux 10.1

2006-05-17T18:08:00.000-04:00

VMWare workstation allows user to run multiple operating systems on one machine. I decided to try the eval version on my newly installed SUSE 10.1 box. The installation however was not as smooth as I would have liked. Installation fails while compiling vmmon module for my kernel version (2.6.16.13-4-bigsmp, SuSE 10.1). Fortunately, there is a workaround as mentioned in this forum, but it took me considerable time to figure out what to do in order to make vmmon compile. Hence, I am posting what exactly needs to be done.

I downloaded the tarball for evaluation version of VMWare workstation from their website and tried to install it using the provided installation script.

tar zxvf VMware-workstation-5.5.1-19175.tar.gz
cd /vmware-distrib/
./vmware-install.pl

I answered all the questions with their default answers. The script after installation executed vmware-config.pl, which exited with the following error while compiling vmmon.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config3/vmmon-only'
make -C /lib/modules/2.6.16.13-4-bigsmp/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-2.6.16.13-4-obj/i386/bigsmp'
make -C ../../../linux-2.6.16.13-4 O=../linux-2.6.16.13-4-obj/i386/bigsmp modules
  CC [M]  /tmp/vmware-config3/vmmon-only/linux/driver.o
In file included from /tmp/vmware-config3/vmmon-only/linux/driver.h:20,
                 from /tmp/vmware-config3/vmmon-only/linux/driver.c:49:
/tmp/vmware-config3/vmmon-only/./include/compat_wait.h:37:5: warning: "VMW_HAVE_EPOLL" is not defined
/tmp/vmware-config3/vmmon-only/./include/compat_wait.h:43:5: warning: "VMW_HAVE_EPOLL" is not defined
In file included from /tmp/vmware-config3/vmmon-only/linux/driver.h:20,
                 from /tmp/vmware-config3/vmmon-only/linux/driver.c:49:
/tmp/vmware-config3/vmmon-only/./include/compat_wait.h:60: error: conflicting types for ‘poll_initwait’
/usr/src/linux-2.6.16.13-4/include/linux/poll.h:45: error: previous declaration of ‘poll_initwait’ was here
/tmp/vmware-config3/vmmon-only/linux/driver.c:145: warning: initialization from incompatible pointer type
/tmp/vmware-config3/vmmon-only/linux/driver.c:149: warning: initialization from incompatible pointer type
make[4]: *** [/tmp/vmware-config3/vmmon-only/linux/driver.o] Error 1
make[3]: *** [_module_/tmp/vmware-config3/vmmon-only] Error 2
make[2]: *** [modules] Error 2
make[1]: *** [modules] Error 2
make[1]: Leaving directory `/usr/src/linux-2.6.16.13-4-obj/i386/bigsmp'
make: *** [vmmon.ko] Error 2
make: Leaving directory `/tmp/vmware-config3/vmmon-only'
Unable to build the vmmon module.

For more information on how to troubleshoot module-related problems, please
visit our Web site at "http://www.vmware.com/download/modules/modules.html" and
"http://www.vmware.com/support/reference/linux/prebuilt_modules_linux.html".

The workaround is to modify the Makefile.kernel, by adding -DKBUILD_BASENAME=\"$(DRIVER)\" just after -Iinclude2/asm/mach-default (on the same line). Do the following step by step.

cd /usr/lib/vmware/modules/source
tar xf vmmon.tar
tar xf vmnet.tar
vim vmmon-only/Makefile.kernel #EDIT as mentioned above
vim vmnet-only/Makefile.kernel #EDIT as mentioned above
mv vmmon.tar vmmon.tar.save
mv vmnet.tar vmnet.tar.save
tar cf vmmon.tar vmmon-only
tar cf vmnet.tar vmnet-only

After modification, first few lines of vmnet-only/Makefile.kernel looks like

vm_check_build = $(shell if $(CC) $(CPPFLAGS) $(CFLAGS) $(CFLAGS_KERNEL) \
        $(EXTRA_CFLAGS) -Iinclude2/asm/mach-default \
        -DKBUILD_BASENAME=\"$(DRIVER)\" \
        -Werror -S -o /dev/null -xc $(1) \
        > /dev/null 2>&1; then echo "$(2)"; else echo "$(3)"; fi)

and that of vmmon-only/Makefile.kernel looks like

After modifying both the Makefile.kernel files, execute /usr/bin/vmware-config.pl again. The compilation should go fine this time.

Apache on RHEL 3 (undefined symbol: gdbm_errno)

2006-05-07T15:51:00.000-04:00

While trying to enable web based admin gui for our sun ray server, I discovered that, httpd on RHEL 3 has some problems with libraries.

[root@manhattan root]# httpd
httpd: relocation error: /usr/lib/libaprutil-0.so.0: undefined symbol: gdbm_errno

Since I had not installed or changed anything after installing RedHat Enterprise Linux AS 3, I believe, this bug comes packaged with the distribution. And I expect Red Hat to test their software more carefully, at least httpd.

To debug, I checked dependencies for libaprutil-0.so.0 and symbols in libgdbm.so. To my surprise, gdbm_errno was in the libgdbm.so, and httpd should not have reported error. Everything (LD_LIBRARY_PATH and ld.so.cache) was fine.

[root@manhattan root]# rpm -aq | grep httpd
httpd-2.0.46-54.ent
redhat-config-httpd-1.1.0-4.30.2

[root@manhattan root]# rpm -aq | grep gdbm
gdbm-devel-1.8.0-20
gdbm-1.8.0-20

[root@manhattan root]# ldd -d /usr/lib/libaprutil-0.so.0
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00cc8000)
        libc.so.6 => /lib/tls/libc.so.6 (0x008c2000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00216000)
undefined symbol: gdbm_errno    (/usr/lib/libaprutil-0.so.0)
undefined symbol: apr_pool_cleanup_null (/usr/lib/libaprutil-0.so.0)

[root@manhattan root]# nm --dynamic /usr/lib/libgdbm.so | grep gdbm_errno
000061c4 D gdbm_errno

After trying out many things and googling, I was able to fix the problem. Before starting httpd, I had to run

[root@manhattan root]# export LD_PRELOAD=/usr/lib/libgdbm.so.2

I am not sure why and how it worked and why exactly was their a problem in the first place. This will probably remain a mystery for me.

Democracy Player on SuSE 10.0

2006-05-02T00:21:00.000-04:00

When I read news about availability of linux version of democracy player, I decided to try it. But unfortunately no SuSE packages or build instructions were available. But, after hours of hard work, I managed to install it on my x86_64 machine running SuSE linux 10.0. Here is what I did.

First I installed Pyrex. Since version 0.9.3 of Pyrex is buggy, I installed v0.9.4.1 by compiling from sources. I also installed mozilla-config from rpm using YaST. Then I installed boost from source (rpm I had, had some problems). To install boost, bjam is required, which fortunately was available as rpm, and I used YaST to install it. After installing bjam, I used the following procedure to install boost.

tar zxvf boost_1_33_0.tar.gz
cd boost_1_33_0
export PYTHON_VERSION=2.4
export PYTHON_LIB_PATH=/usr/lib64/python2.4/
export PYTHON_INCLUDE_PATH=/usr/include/python2.4/
sudo bjam "-sTOOLS=gcc" install

I then proceeded to install Democracy player from source. Democracy source must reside in directory named tv. Also libboost_python-gcc-mt-1_33.so.1.33.0 had to be manually linked to libboost_python.so. Since boost was installed in /usr/local/, I also had to set appropriate CPPFLAGS and LD_LIBRARY_PATH.

tar zxvf Democracy-0.8.2-Source.tar.gz
mv Democracy-Player-0.8.2/ tv
cd tv/platform/gtk-x11/
export DEMOCRACY_RESOURCE_ROOT=/usr/share/democracy/resources/
export CPPFLAGS="-I/usr/local/include/boost-1_33/"
export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH
export PATH=/opt/mozilla/bin/:$PATH
sudo ln -s /usr/local/lib/libboost_python-gcc-mt-1_33.so.1.33.0 /usr/local/lib/libboost_python.so
python setup.py install --root=/

This installed Democracy player, but I still wasn't able to start it, because gtkmozembed was not available. I then had to compile gnome-python-extras-2.12.1 from source.

tar zxvf gnome-python-extras-2.12.1.tar.gz; cd gnome-python-extras-2.12.1
./configure --prefix=/usr/
make
sudo make install

Now to start Democracy Player, I need to

cd tv/platform/gtk-x11/
export DEMOCRACY_RESOURCE_ROOT=/usr/share/democracy/resources/
export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH
export PATH=/opt/mozilla/bin/:$PATH
python democracyplayer

After installation, I started Democracy and downloaded a few videos (which were saved to ~/Movies/Democracy/). While videos played fine, there was no sound. I tried playing videos with mplayer they worked fine. The problem was with xine, and I had to install xine from source again. After all this, Democracy is now working fine (except that it has loads of bugs and is very unstable).

Moving a Folder Across SVN Repositories

2006-03-04T13:05:00.000-05:00

Sometimes we may want to transfer a directory from one subversion repository to another without losing any revision history. This can be dones using svnadmin dump and load with svndumpfilter. The following commands will create the directory /folder/to/move/ in the target repository.

svnadmin dump /path/to/sourcerepo/ | svndumpfilter include /folder/to/move/ > dumpfile
svnadmin load /path/to/targetrepo/ < dumpfile

This creates the folder /folder/to/move/ in targetrepo, and there is no simple way (except probably by editing the dumpfile manually) to instruct it to create only the folder /move/.

svndumpfilter does not filter out empty revisions. Hence, one may like to add options --drop-empty-revs --renumber-revs to svnadmin load to drop empty revisions and adjust revision numbers when an empty revision is dropped. More information about administration utilities in svn can be found at the red book.

Google Talk voice chat on SuSE

2006-01-21T19:18:00.000-05:00

One of the things that I sometimes miss on Linux is the ability of voice chat. Skype is the only IM network that supports voice call to and from Linux machines. However, with recent release of libjingle by Google, it is now possible to call my friends using Google Talk from Linux. psi-jingle branch of psi has implemented the support of libjingle. I recently installed it on my desktop running SuSE 9.3 and it works great. Here is the procedure for installation.

1. Get the sources either directly from the darcs repository

darcs get --partial --set-scripts-executable http://dev.psi-im.org/darcs/psi-jingle

or download this tarball.

2. Install all required packages (use apt)

apt install speex-devel
apt install qca-devel

3. Compile ORTP from source because the version of package in repository was not compatible.

wget http://www.linphone.org/ortp/sources/ortp-0.7.1.tar.gz
tar zxvf ortp-0.7.1.tar.gz; cd ortp-0.7.1
./configure --prefix=/usr/
make && make install

4. Install psi-jingle

cd psi-jingle
./configure --enable-jingle --with-qca-inc=/usr/lib/qt3/include/ --with-qca-lib=/usr/lib/qt3/lib/   
           --with-ortp-lib=/usr/lib/ --with-ortp-inc=/usr/include/  
           --with-glib-inc=/opt/gnome/include/glib-2.0/ --with-glib-lib=/opt/gnome/lib/ 
           --with-glibconfig-inc=/opt/gnome/lib/glib-2.0/include/
make

5. Fire up psi and login to Google Talk network and start voice chatting.

It is important to note that psi-jingle is an under development code and there may be bugs (although it works fine for me). If there are problems, check artsd is not running and mic is functioning fine.

Note: these instructions will work only with 32-bit OS. While I was able to successfully compile it on amd64, the procedure was non-trivial and required me to edit many Makefiles and source files manually.

FreeNX on RHEL 3

2006-01-19T15:20:00.000-05:00

RPMs referred here are available at http://www.db.toronto.edu/~nilesh/linux/rpms/RHEL3/.

1. Install NX and FreeNX rpms

rpm -ivh --nodeps nx-1.5.0-12.el3.at.i386.rpm
rpm -ivh --nodeps freenx-0.4.4-6.el3.at.noarch.rpm

2. Setup everything

nxsetup --install --setup-nomachine-key --clean --purge

3. To start/stop use

nxserver --start
nxserver --stop
nxserver --status

Optionally create /etc/init.d/nxserver with following content

# File /etc/init.d/nxserver
# RHEL startup script
[ -f /usr/bin/nxserver ] || { echo "ERROR: /usr/bin/nxserver doesn't exist" ; exit 1; }

function start()
{
  /usr/bin/nxserver --start
  /usr/bin/nxserver --statistics start
}
function stop()
{
  /usr/bin/nxserver --stop
  /usr/bin/nxserver --statistics stop
}
function restart()
{
  /usr/bin/nxserver --restart
  /usr/bin/nxserver --statistics restart
}
case $1 in
start)
      start
   ;;
stop)
    stop
  ;;
restart)
    restart
  ;;
*)
  echo "Usage: $0 {start|stop|restart}"
  exit 1
  ;;
esac

and run the commands

/etc/init.d/nxserver restart
chkconfig nxserver on

Media Wiki on RHEL4

2006-01-19T14:49:00.000-05:00

To install MediaWiki on RedHat Linux Enterprise Edition, follow thw following instructions:

1. Download MediaWiki from http://www.mediawiki.org/wiki/Download and untar it

tar zxvf mediawiki-1.5.5.tar.gz
mv mediawiki-1.5.5/ /var/www/wiki

2. Check that httpd and php are installed (from the default installation). I have the following rpms installed.

httpd-2.0.52-19.ent
php-4.3.9-3.9

3. Install MySQL and php-mysql. I had following RPMs.

MySQL-shared-standard-4.1.16-1.rhel4
MySQL-server-standard-4.1.16-1.rhel4
MySQL-devel-standard-4.1.16-1.rhel4
MySQL-client-standard-4.1.16-1.rhel4
php-mysql-4.3.9-3.9

4. Create account on MySQL for media wiki. Either use mysql-administrator or phpMyAdmin or the following commands

[root@queens]# mysql -uroot -hlocalhost -p mysql
mysql> create database wikidb;
mysql> GRANT ALL PRIVILEGES ON wikidb.* to 'wikiuser'@'%' identified 
       by 'wikipasswd' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;

5. Change the permissions of MediaWiki config directory.

chmod a+w /var/www/wiki/config/

6. Visit the wiki URl (http://yourhost/wiki/) and enter all configuration details. When finished move LocalSettings.php to parent directory and change permissions of config directory.

mv /var/www/wiki/config/LocalSettings.php /var/www/wiki/
chmod -R og-w /var/www/wiki/config/

All RPMS are available at http://www.db.toronto.edu/~nilesh/linux/rpms/RHEL4/.

Subversion HOWTO

2006-01-05T03:12:00.000-05:00

Recently I installed subversion with trac by compiling Apache and everything. I have written a howto explaining the process in detail.

http://www.cs.toronto.edu/~nilesh/linux/subversion-howto/

Security is Important

2005-12-28T03:03:00.000-05:00

A few days back I configured apache on one of the machines and made it visible to outside world. No content was uploaded and I expected no one to visit the page. Today, when I was checking the access_log for apache (just out of curiosity), I noticed repeated requests for xmlrpc.php (although there was no such file on my machine). There were atleast 12 different machines (in just 3 days) scanning the host for xmlrpc.php. A quick google search revealed the vulnerability in xmlrpc which these machines were trying to exploit.

I always ignored security advisories, and considered them only for paranoids. But similar incidents in last few days have made me realize that security is a real problem. There are so many "bad" machines out there, scanning every other machine for some vulnerability. There is a whole economy behind these viruses and spam. It is therefore important to be cautious.

Plotting in Matlab without X-window

2005-11-25T00:17:00.000-05:00

Matlab can be operated in non-GUI mode. To start matlab:

matlab -nodesktop -nojvm -nosplash

If you wish to plot graphs without using X or GUI, open a figure with visibility hidden, plot the graph and then print the figure to a file.

>> h = figure('Visible', 'off')
h =

     1

figure returns a handle to figure. Now issue the plot command.

plot(1:4,5:8)

The figure can now be saved using the print command.

print -f1 -dps 'filename.ps'

The syntax for the print command is
print -fhandle -ddevice.
In our case, the handle is 1 and device is post script file.

Update: As posted by anonymous, another option is to use

saveas(h,'file.png');

New Blog

2005-11-07T22:34:00.000-05:00

I now have this space where, hopefully, I will be writing something.