How to run a shell script from a Java application

Introduction

Running a shell script from a Java program using Runtime.exec() seems superficially simple. Yet, time and again, I see developers struggling to get it to work reliably. The Java documentation does not provide a lot of help, even though the exec() method has had developers banging their heads on their desks for as long as Java has existed.

Runtime.exec() is problematic for reasons that have nothing to do with running shell scripts — scripts simply add a bunch of other pain points to an already painful experience.

This article describes in detail why exec() and its related API methods are so difficult to use properly, and outlines what I hope is a reasonably robust implementation.

I've tested the steps I describe on Linux, although I've no particular reason to think they won't be applicable to other Unix-like systems. The basic principles apply to Windows, too but, of course, the use of the command-line interpreter is completely different.

The basic problem

From a Java application, I want to run the shell script something.sh, and pass it the string 42 as an argument. That is, I want to get the same effect I would get if I executed at the shell prompt:

$ something.sh 42

Specific problems with running scripts

The problematic implementations I typically see look something like this:

   Process p = Runtime.getRuntime().exec ("something.sh 42");
   int e = p.exitValue();
   System.out.println ("Exit code = " + e);

I'll explain first what's wrong with this implementation, specifically as a way of running a script. Then I'll describe what would be wrong with it as a way of running any kind of executable.

exec() (probably) only handles binaries

Most fundamentally, something.sh is not an executable binary — it is, of course, a shell script. Although we get used to running scripts and binaries interchangeably on the command line, they aren't the same thing at all. To run a script, we must run a shell and indicate the location of the script.

I should point out, in the interests of completeness, that some Linux implementations are configured to be able to run scripts at the system call level. That is, the kernel can actually process a script as if it were an executable, by implicitly invoking a shell. However, this isn't common behaviour, and shouldn't be relied on.

To invoke the shell to run the script, we might try something like this:

   Process p = Runtime.getRuntime().exec ("sh -c something.sh 42");

The -c switch tells the shell to process the next argument as a command. This won't work, either, because ...

exec() does not know how to group argments

In my example, I want to invoke something.sh and pass it the argument 42. But Runtime.exec() doesn't know that my "42" is an argument to "something.sh"; it thinks that all three arguments are for sh. So something.sh might get invoked, but it won't get the 42 argument. Or, indeed, any arguments.

Perhaps we can use quotes to group the arguments? This won't work, either, because...

The string tokenization done by exec() is crude

It looks as is this should work:

   Process p = Runtime.getRuntime().exec ("sh -c \"something.sh 42\"");

That is, I've enclosed the arguments to sh -c in double-quotes, to indicate to the shell that there is a single argument to something.sh.

However, it's not the shell we need to be concerned about here — it's the crudeness of the exec() string tokenization. The method exec(String) simply splits the string on whitespace. It doesn't understand quotes or escapes.

So, in this case, the sh utility still gets three arguments (four, including 'sh' itself). The third argument is "something.sh, and the fourth 42". In practice, it's going to be necessary to tokenize the string ourselves. That may be easy or it may be difficult, depending on the application's needs. In any case, we need to end up with an array of String instances that can be passed to exec (String[]).

This should do the trick in this simple case, where we know all the arguments ahead of time:

   Process p = Runtime.getRuntime().exec
        (new String[]{"sh", "-c", "something.sh 42"});

Here we've explicitly passed a single argument after sh -c — the script and its arguments — to be processed by the shell. But we're still not done.

The fundametal problem raised in the last few sections is that Runtime.exec() is not a command-line processor — it's only a process launcher. As well as not tokenizing very intelligently, it won't expand shell wildcards (*.txt), or substitute environment variables ($HOME). If you're launching a shell script, you should expect to have to do any of this kind of thing in the script itself.

Be careful about file locations

In practice, the shell might need to be told where to find the script something.sh. On Linux the shell won't necessarily look in the current directory, although it might look in the directories specified in $PATH. If the script is in the working directory of the Java application, you probably need to do this:

   Process p = Runtime.getRuntime().exec
        (new String[]{"sh", "-c", "./something.sh 42"});

Alternatively, you could use the full pathname of the script.

Other things to watch out for, when running a script

It should be obvious, but bear in mind that running a script using Java doesn't do any additional magic. The script still needs to have execute permissions to be run as a command, and it will usually still need to start with an interpreter line:

#!/bin/sh

These things aren't specific to Java — my point is that using Java doesn't mean that we can stop thinking about basic platform issues.

General pitfalls, not related to scripts

The preceeding sections described pitfalls in the use of Runtime.exec() related specifically to executing shell scripts. However, we haven't yet even touched on the pitfalls that apply to any use of exec().

exec() does not block

The exec() method starts a process, but does not wait for it to complete. In our example the shell script might run perfectly well, but the Java application will fail at the call to exitValue(). If the script has not yet finished by the time this method is called — and most likely it will hardly have started — then the exitValue method will throw IllegalThreadStateException.

If we want the Java application to wait for the script to complete, we must code this explicitly, like this:

   Process p = Runtime.getRuntime().exec ("something.sh 42");
   p.waitFor();
   int e = p.exitValue();

Be aware that, in principle, waitFor() can throw an InterruptedException. To be frank, this is unlikely — profoundly unlikely in a single-threaded application — and it's difficult to know how to handle the exception if it is raised.

This brings me to the biggest bugbear of running processes from Java — the need to handle process output.

The application must handle standard out and standard error

The script (or other process) being executed may produce output, to standard out or standard error. If it does, and that output is not read, then the process being executed will block forever. Depending on how the Java application is coded, you may or may not notice this stalled process.

On some Linux versions, the process will block even if it doesn't produce output. It's never safe to assume that you don't need to read standard out and standard error. In any event, it's almost always good practice to collect this data, if only for troubleshooting why the script didn't work.

Because you need to collect both standard out and standard error at the same time (unless you can absolutely certain that only meagre amounts of output are generated), you'll need to create an additional two threads, just to read and process the output. One of these threads will read standard out, the other standard error. The threads need to run until the streams to which they are attached close.

To get the relevant streams, do this:

  Process p = Runtime.getRuntime().exec
        (new String[]{"sh", "-c", "./something.sh 42"});
  InputStream stdout = p.getInputStream();
  InputStream stderr = p.getErrorStream();

I'm all too aware how confusing it can be, that the output from the process is an InputStream. From the point of view of the Java program, however, it is a source of data, not a sink, and thus InputStream is entirely appropriate.

The streams provided by Process.getInputStream(), etc., should be supplied to threads, which will then consume the streams, and process the results according to the application's needs. In some cases it will be sufficient simply to accumulate the process output, if there is any, and log it somewhere. In more complex cases the data will need to be parsed and processed.

Here is the outline of a class that implements a thread to consume the output of the launched process.

class StreamEater extends Thread
  {
  BufferedReader br;

  /** Construct a StreamEater on an InputStream. */
  public StreamEater (InputStream is)
    {
    this.br = new BufferedReader (new InputStreamReader (is));
    }

  public void run ()
    {
    try
      {
      String line;
      while ((line = br.readLine()) != null)
        {
	// Process the line of output in some way
        }
      }
    catch (IOException e)
      {
      // Do something to handle exception
      }
    finally
      {
      try { br.close(); } catch (Exception e) {};
      }
    }
  }

Having obtained the standard out and standard error streams from the Process, they can be passed to the stream-consuming threads, like this:

  StreamEater stdoutEater = new StreamEater (stdout);
  StreamEater stderrEater = new StreamEater (stderr);
  stdoutEater.start();
  stderrEater.start();
  p.waitFor();

When waitFor() returns, the launched process will have completed, and the threads created to process the output will have finished their run() methods. However, the thread objects will still exist, and any data that was accumulated during the running of the process will continue to be available.

It's impossible to give general advice on how to handle the output of the process, or whether it's best to use a single stream consumer to handle both stdout and stderr. If the process generates only a little output (kilobytes to megabytes), it might be practical to buffer it all up and process it after the process completes. In more complicated cases, the stream consumer threads will need to process the data as it is delivered to them.

Closing remarks

Running a process — particularly a shell script — from a Java application is surprisingly difficult to do robustly. A complete example, based on the description above, is available from my GitHub repository.