Monday, July 23, 2007

 

Accessing files over SFTP in Java

We have a lot of data files that needs to be copied to each of the machine we want to run our code on. Further when these data files are updated, they need to be updated on all the machines. This means that the developer has to spend a lot of time just copying these data files around. I wrote a simple solution to this where latest versions of all the data files are maintained at a central server accessible via ssh. These remote date files are copied to the local machine when required in an on-demand fashion transparently by the Java program (after comparing last modification times of the local and remote file).

For accessing files over SFTP, we are using Apache Commons VFS along with Jsch. These libraries (especially commons VFS) is not well documented. I am therefore posting some code snippets from our code documenting the API

The first code snippet demonstrates the API for copying a file from remote location to the local machine:


/**
 * Copies a remote file to local filesystem.
 */
public static void copyRemoteFile(String host, String user,
    String password, String remotePath, String localPath) throws IOException {
    // we first set strict key checking off
    FileSystemOptions fsOptions = new FileSystemOptions();
    SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
            fsOptions, "no");
    // now we create a new filesystem manager
    DefaultFileSystemManager fsManager = (DefaultFileSystemManager) VFS
            .getManager();
    // the url is of form sftp://user:pass@host/remotepath/
    String uri = "sftp://" + user + ":" + password + "@" + host
            + "/" + remotePath;
    // get file object representing the local file
    FileObject fo = fsManager.resolveFile(uri, fsOptions);
    // open input stream from the remote file
    BufferedInputStream is = new BufferedInputStream(fo.getContent()
            .getInputStream());
    // open output stream to local file
    OutputStream os = new BufferedOutputStream(new FileOutputStream(
            localPath));
    int c;
    // do copying
    while ((c = is.read()) != -1) {
        os.write(c);
    }
    os.close();
    is.close();
    // close the file object
    fo.close();
    // NOTE: if you close the file system manager, you won't be able to 
    // use VFS again in the same VM. If you wish to copy multiple files,
    // make the fsManager static, initialize it once, and close just
    // before exiting the process.
    fsManager.close();
    System.out.println("Finished copying the file");
}

Unfortunately the Commons VFS api does not provide a way to check last modification time of a remote file. I had to write that code using the Jsch API. Below is a code snippet that returns last modification time in seconds:


/**
 * Returns a Sftp session conncted using the Jsch library.
 */
public static Session connectSFTP(final String host, final String user,
        final String pass) throws JSchException {
    JSch jsch = new JSch();
    Session session = jsch.getSession(user, host, 22);
    session.setUserInfo(new UserInfo() {
        public String getPassphrase() {
            return null;
        }
        public String getPassword() {
            return null;
        }
        public boolean promptPassphrase(String string) {
            return false;
        }
        public boolean promptPassword(String string) {
            return false;
        }
        public boolean promptYesNo(String string) {
            return true;
        }
        public void showMessage(String string) {
        }
    });
    session.setPassword(pass);
    session.connect();
    return session;
}

/**
 * Returns last modification time of a remote file in seconds.
 */
public static int getLastModificationTime(String host, String user,
        String password, String remotePath) throws IOException,
        JSchException, SftpException {
    Session session = connectSFTP(host, user, password);
    ChannelSftp chan = (ChannelSftp) session.openChannel("sftp");
    chan.connect();
    SftpATTRS attrs = chan.lstat(remotePath);
    int time = attrs.getMTime();
    chan.disconnect();
    session.disconnect();
    return time;
}

I hope that this code is useful to others. Please leave a comment if you see any error of have a suggestion.

Labels: ,


Comments:
for a well documented sftp library you might look at secure ftp factory

http://www.jscape.com/sftp/
 
@richard The sftp library you are mentioning is not Open Source and only a 30 day evaluation is available for free.
 
which is why it is well documented! ;-)
 
Thanks for the code. Using this as a start I was able to write a short app to move files. Thanks
 
use rsync utility for copying files from server to other
 
hi nilesh...can u post me a java code to transfer a file as it is from local machine to a remote machine???

ujjal
 
thanks alot for the code. I'd been searching the web for quite a while for something like this
 
thanks for your code. I could not get your sftp to work, but using Jsch classes and using their sample exec.java program wrote code to use scp to file transfer.

Is there a way to share this code, and comments?

thanks
 
For another well documented SFTP library take a look at edtFTPj/PRO

http://www.enterprisedt.com/products/edtftpjssl/overview.html

Again, it is not open source but it includes neat features like connection pooling and asynchronous operations.

Also supports FTP and FTPS. We do have a very popular open source FTP library too which this is built on - edtFTPj.
 
This doesn't work with larger files. The VFS seems to have an issue with the getInputStream(), where it loads the whole file into memory before returning a memory stream. Eventually the JVM will run out of memory. In any case, it was a nice start--I'm going to try using Jsch to work around this problem.
 
Yes. The code is not working with larger files. It's OK upto 30mb files. If it exceeds more than 30mb it's causing problems(memory issues). Any idea how to get rid of this problem.
 
I used a straight Jsch implementation. The Apache virtual file system stuff is catchy, and certainly easy to use, but that major fault for uploading/downloading large files makes it unusable for some applications.
 
You can try SFTP applet that comes with resume support, upload and file manager.
 
I think that you have a comment that says

// get file object representing the local file

but it should really read

// get file object representing the remote file

Thank you for the code!
 
I also use this SFTP API from http://www.zehon.com. It's free and easy to use too.
 
The latest nightly builds of Commons VFS have fixed the problems with large files.
 
Unfortunately doesn't work with @ at passwords.
 
Where can I download the nightly builds ?
 
I haven't confirmed that it works, but you can get it from here:
http://people.apache.org/builds/commons/nightly/commons-vfs/
 
Thanks for your response. There I already was... The date seems weird to me (....2007).
 
JSch.setConfig("StrictHostKeyChecking", "no");

This is necessary to dont ask about the Know Host
 
This comment has been removed by a blog administrator.
 
plug: sshj (http://github.com/shikhar/sshj) has a complete implementation of sftp and a nice api. it's young so i'm hoping to get users who will report bugs :)
 
I found good working example for SFTP in java at http://vigilance.co.in/java-program-for-uploading-file-to-sftp-server
 
For whoever is too lazy to create the version that does the upload, here you go:

public static final void copyLocalFileToRemote(String host, String user, String password, String remotePath, String localPath) {
FileObject fo = null;
BufferedInputStream is = null;
OutputStream os = null;

// we first set strict key checking off
FileSystemOptions fsOptions = new FileSystemOptions();

try{

SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fsOptions, "no");
// now we create a new filesystem manager
DefaultFileSystemManager fsManager = (DefaultFileSystemManager) VFS.getManager();
// the url is of form sftp://user:pass@host/remotepath/
String uri = "sftp://" + user + ":" + password + "@" + host + "/" + remotePath;
// get file object representing the remote file
fo = fsManager.resolveFile(uri, fsOptions);

// open input stream from the local file
is = new BufferedInputStream(new FileInputStream(localPath));
// open output stream to remote file
os = new BufferedOutputStream((fo.getContent().getOutputStream()));

int c;
// do copying
while ((c = is.read()) != -1) {
os.write(c);
}
os.close();
is.close();
// close the file object
fo.close();

}catch(IOException ex){
System.out.println("SEVERE (convert to log4j) Error -- Caught IOException: ");
ex.printStackTrace();
}finally{
/* No need to worry about empty catch blocks, just cleaning up incase */
try{
if(fo != null)
fo.close();
}catch(Exception ex){}

try{
if(is != null)
is.close();
}catch(Exception ex){}


try{
if(os != null)
os.close();
}catch(Exception ex){}

}
}
 
hi nilesh/ujjal... Please give me the code to upload a file to a sftp location.
 
Also this program dosn't work when I say session.setProxy(
Can you please give an eg of JSchOverJHttpTunnel
 
Good blog.. I was looking for a code to get the last modified time of the file.
 
good blog.. I was looking for a code to get the modified timestamp of remote file.
 
I am getting the below error when the username or password has an escape character such as "%" or ":" . Instead of conatenating strings to build the URI is there a better way to build the URI encoded?

: Invalid URI escape sequence "%ph".
org.apache.commons.vfs2.FileSystemException: Invalid URI escape sequence "%gf".
at org.apache.commons.vfs2.provider.UriParser.decode(UriParser.java:343)
at org.apache.commons.vfs2.provider.UriParser.decode(UriParser.java:308)
at org.apache.commons.vfs2.provider.UriParser.checkUriEncoding(UriParser.java:462)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:678)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:621)

 
Post a Comment

Links to this post:

Create a Link



<< Home

This page is powered by Blogger. Isn't yours?