The iDRAC Virtual Console uses Java Web Start (JWS), which was deprecated in Java 9, and starting with Java 11, Oracle completely removed JWS from their JDK distributions.
Below you will find instruction on how to get everything working again with the help of OpenWebStart.
Install OpenWebStart from the link above.
System Preferences > Security & Privacy > Privacy and in
Open the iDRAC web interface and navigate to:
System > Properties > System Summary.
Virtual Console Preview widget go to
Options: Settings and uncheck
Video Encryption Enabled.
Navagate back to
System > Properties > System Summary and click Launch on the on the
Virtual Console Preview widget, this will download a
jpnl file that you should rename each time you download it to
viewer.jpnl or something similar.
Right click the file and select
Get Info, set
Open with to
OpenWebStart javaws and click
change all. This will make sure the jpnl files are always opened with OpenWebStart.
Right click again and this time open the file to start the virtual console.
If you get the message
Connection failed. run the
sed command below, it will remove
RC4, what the viewer needs, from the disabled TLS algorithm list in the java.security configuration of all Java VM's in
sed -i 's/jdk.tls.disabledAlgorithms=\(.*\) RC4,\(.*\)/jdk.tls.disabledAlgorithms=\1\2/' ~/.cache/icedtea-web/jvm-cache/*/conf/security/java.security
That is it to get the virtual console working.