Fix Zabbix Java Gateway with a custom Java version (error "compiled by a more recent version of the Java Runtime")

In a production environment we have a simple Zabbix monitoring system that works very well to collect metrics from various Tomcat instances and other Java applications.

In short, this is our current configuration (that is somehow classic):

[ Zabbix Server ] → [ Zabbix Agent (passive) ] → [ Zabbix Java Gateway ] → [ Tomcat ]

It works also in this way:

[ Zabbix Server ] ← [ Zabbix Proxy ] →  [ Zabbix Agent (passive) ] → [ Zabbix Java Gateway ] → [ Tomcat ]

The above configuration works out of the box for us with any native package in the following distributions at least:

• Debian GNU/Linux 11 (current stable) (https://packages.debian.org/bullseye/zabbix-java-gateway)
• Ubuntu 20.04 (current LTS) (https://packages.ubuntu.com/focal/zabbix-java-gateway)
• Ubuntu 22.04 (current LTS) (https://packages.ubuntu.com/jammy/zabbix-java-gateway)

This is an example of how we suggest to install the agents:

sudo apt install zabbix-agent zabbix-java-gateway

The problem

For application reasons we now need a custom Java JDK. But, that is not compatible with the Zabbix Java Gateway.

For example if we configure one Java JDK in a non-standard place, to avoid any possible conflict with existing packages, and if then we set that as default, like in this way:

update-alternatives --install /usr/bin/java java /opt/jdk-production/bin/java 2222
update-alternatives --config java

From that point, the systemd daemon of zabbix-java-gateway stops working.

This is the following stack trace:

$ journalctl -fu zabbix-java-gateway
Apr 27 16:09:22 example-hostname systemd[1]: Starting Zabbix Java Gateway...
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]: Error: A JNI error has occurred, please check your installation and try again
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]: Exception in thread "main" java.lang.UnsupportedClassVersionError: com/zabbix/gateway/JavaGateway has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.lang.ClassLoader.defineClass1(Native Method)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.security.AccessController.doPrivileged(Native Method)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[956]:         at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:601)
Apr 27 16:09:22 example-hostname zabbix_java_gateway[947]: Zabbix Java Gateway did not start
Apr 27 16:09:22 example-hostname systemd[1]: zabbix-java-gateway.service: Control process exited, code=exited, status=1/FAILURE
Apr 27 16:09:22 example-hostname systemd[1]: zabbix-java-gateway.service: Failed with result 'exit-code'.
Apr 27 16:09:22 example-hostname systemd[1]: Failed to start Zabbix Java Gateway.

So, the package is clearly not compatible with our JDK based on Java 8.

In short, we would like to:

• just fix the Zabbix Java Gateway
• possibly, avoid to install non-native packages (avoid to install another Zabbix Java Gateway)
• possibly, keep our custom Java version as the default one

Question

What is the Debian-way or Ubuntu-way to fix similar issues with the Zabbix Java Gateway?

What is happening

The error is obviously normal since the native zabbix-java-gateway is compiled with a recent version of Java, and you are adopting a default legacy version of Java.

How to Fix (with minor changes, keeping your custom JDK)

You can keep your custom Java JDK as default, and fix this.

Fortunately, the package zabbix-java-gateway supports a simple way to adopt whatever java version. Also, you do not even need to install an extra Java version, since you already have installed the needed OpenJDK as dependency of Zabbix Java Gateway.

You just need to force the Zabbix Java Gateway to adopt OpenJDK again, just for that executable.

Create this file:

/etc/default/zabbix-java-gateway

And put a similar content:

JAVA=/opt/path-to-your-jdk/java

For example for Ubuntu 22.04 LTS, this is a valid content for that file:

JAVA=/usr/lib/jvm/java-11-openjdk-amd64/bin/java

If you are not sure about the correct Java path of your OpenJDK, double-check with this command:

update-alternatives --list java | grep openjdk

At this point just restart your service:

sudo systemctl restart zabbix-java-gateway

And everything should be OK.

In this way, you keep your custom JDK, but Zabbix Java Gateway uses the native one.

And now just a note about why that ↑ works.

This works since the systemd daemon of the Zabbix Java Gateway works in this way (here from Ubuntu 22.04 current LTS):

$ systemctl cat zabbix-java-gateway
# /etc/systemd/system/zabbix-java-gateway.service
[Unit]
Description=Zabbix Java Gateway
After=syslog.target
After=network.target

[Service]
Type=forking
KillMode=process
PIDFile=/run/zabbix/zabbix_java_gateway.pid
ExecStart=/usr/sbin/zabbix_java_gateway
SuccessExitStatus=143
User=zabbix
Group=zabbix

[Install]
WantedBy=multi-user.target

And if you inspect the executable mentioned in ExecStart, it has this internal logic:

...
if [ -r "/etc/default/zabbix-java-gateway" ]; then
        . /etc/default/zabbix-java-gateway
fi
...

JAVA=${JAVA:-java}

...

COMMAND_LINE="$JAVA 

...

exec $COMMAND_LINE

So this really means the Zabbix Java Gateway does this at startup:

1. execute the file /etc/default/zabbix-java-gateway if available
2. assume the variable $JAVA to "java" as default - since the meaning of that bash line is: NEWVAR=${VARNAME:-defaultvalue}
3. execute $JAVA ("java") as default

In short,

That is why you can just edit the file /etc/default/zabbix-java-gateway. I thank the package maintainer for this native package done so well for Debian and Ubuntu and that allows this minimal solution.