13

This will be a network application that will always (or near as always as I can manage) be listening on a given port.

I'm fairly new to Java, and very new to non-web server side programming, so I'd like to get feedback from the community on my assumptions and preliminary plans.

I've read about jsvc ( http://commons.apache.org/daemon/jsvc.html ) and am currently operating on the assumption that this is the "best" way to write a daemon in java for a linux box (likely running centOS).

Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't? (I assume yes, but I'm not a very talented sys admin type)

This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP. However, I know that it's more "traditional" to write a daemon in C/C++. With modern Java, performing fairly uncomplicated tasks, am I likely to run into any major disadvantages?

What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?

Any other input would be greatly appreciated.

Improve this question
6
  • 1
    The jsvc doc says that it allows the application (like Tomcat) to perform some application as root (like bind to a port < 1024) and then lower the app's privileges. I honestly don't like allowing to run as root. The way I bind to a port < 1024 is by running as non-root and doing a firewall redirection, like this: "iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080" Note that you can also use such a trick deploy a new daemon without needing to "stop the daemon and replace the binary as quickly as possible" [sic].... (to be continued) Commented Oct 18, 2011 at 22:12
  • 1
    If you're tied to a unique machine, you could use the following trick: leave the "old" daemon running (say on port 8080, which 80 redirects to) then deploy a new daemon (say on port 8081). Test that everything works fine and once you're sure your new daemon is good to go, simply change the port-redirection 80->8080 to 80->8081. The only time root is required for such a thing is to change the firewall redirections. On Unix-like systems, you can install Java and Tomcat without ever needing to log in as root (as long as you don't use any .deb or .rpm packages). Commented Oct 18, 2011 at 22:15
  • I think you're right about jsvc, and your suggestion for redirecting ports is awesome and I will likely do that. Thanks! Commented Oct 19, 2011 at 4:09
  • wait, don't get me wrong: I'm not criticizing jsvc (I don't know it). I'm just saying that the part that says it helps run Java things as root, I'm not big fan of :) It may be doing other very useful things. But the port redirection thing works for sure and if you're in control of the machine, it's really easy and convenient and relatively "safe" (at least you don't need to install Java as root). Commented Oct 19, 2011 at 12:36
  • In a reply the OP says "My question is vague enough that there's no "real" answer" Commented Oct 21, 2013 at 16:41

2 Answers 2

Reset to default
23

How to write a Java daemon that has 24/7 uptime...

We run a number of 24/365 applications on our Linux servers that just call the Java like the following -- no need for any C wrappers:

nohup java -D... -X... -jar something.jar ... < /dev/null > output.log 2>&1 &

That will put the jar running in the background (nohup ... &) with no input (< /dev/null) and the output (stdout and stderr) redirected to a logfile (> output.log 2>&1). We have distributed logging infrastructure but some console output (such as thread dumps) is still expected. These applications can run for months until we upgrade them.

Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't?

In terms of monitoring, there is much you can do. Nagios looks to have a JMX plugin for testing the information that jconsole displays. There are also a lot of native JMX logging and monitoring utilities out there. We have internal green/yellow/red indicators that can be pulled up using JMX and easily checked. I also have exported a simple JMX/HTTP service from each application to provide status information making it easy for 3rd party monitoring tools to detect failures.

This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP.

I assume you mean SMPP? If so then I see no reason why Java couldn't do a good job. Our applications do a wide variety of HTTP, UDP, SMTP, JDBC, LDAP, and other protocols in real time. We use Jgroups at lot which accomplishes a complete authenticated, encrypted, network stack in Java.

What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?

In terms of replacing a running binary on the fly, that it more complicated. We have VIPs up front and replace the binaries at our leisure. Our internal protocols are designed to failover. If you do not have a VIP then one thing to consider would be an orderly handoff. You boot the new jar and it talks to the application running the old jar when it is ready to bind to the new port. Then the old application unbinds and the new one binds immediately afterwards. Something like that.

Hope this helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

My question is vague enough that there's no "real" answer, but this post is awesome and helpful, so check-mark you get!
Can some one please describe the Linux script written ?
I tried using this method. the application starts if I execute the command directly on console or in a bash script but if I try to execute via sudo systemd start app.service the application does not start. /var/log/syslog has some messages saying application started/stopped but don't see anything.
7

If you really want to have something running non-stop on *nix, I recommend you have a look at daemontools.

There are some examples on how to do this here and here.

Basically svscan will spawn a process that monitors your java process from init, and every time it crashes, it gets restarted.

Improve this answer

5 Comments

Java programs really don't "crash" @krico. They can get VM faults but that happens extremely rarely in my experience. They can, however, run out of memory and become unresponsive depending on memory leaks or other application issues. In that case I'm not sure like daemontools would help. I think a more active monitor which pings the application is in order.
@Gray. While in development they don't, but, once in production, they do. to krico +1 daemontools code is ugly and old, but it still works.
Huh. I've run java applications in a cluster in production for years with maybe a handful of VM faults. Guess it depends on the JVM / OS.
Just talked to my ops guy. Back of the napkin calculation is that we have accumulated 900+ months of 24/7 java application runtime in production (~18 servers, 4+ years) and he can remember 0 (zero) times that the VM faulted. I'm pretty sure I remember 1 time but it might have been a kernel panic. Again, we have had out-of-memory and other application faults but very few (if any) VM "crashes".
@Gray: It is quite rare for VMs to fault, but daemontools is already helpfull just for those rare cases. My approach sometimes is to actually force a shutdown in case of an unrecoverable exception, and count on daemontools to bring the app back to life. Very simple example would be catch(OOM e){ System.exit(-1); }.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.