Failing to start up server and issues with 4064

Our Old Omero Server stopped working and we were no longer able to log in even though the webpage comes up.
The admin diag. is below

================================================================================
OMERO Diagnostics 5.1.2-ice35-b45
================================================================================


ERROR:omero.util.UpgradeCheck:<urlopen error [Errno -3] Temporary failure in name resolution>
Commands:   java -version                  1.7.0     (/usr/bin/java)
Commands:   python -V                      2.6.6     (/usr/bin/python)
Commands:   icegridnode --version          3.5.1     (/usr/bin/icegridnode)
Commands:   icegridadmin --version         3.5.1     (/usr/bin/icegridadmin)
Commands:   psql --version                 9.3.5     (/usr/bin/psql)

Server:     icegridnode                    not started

Log dir:    /home/omero/OMERO.server-5.1.2-ice35-b45/var/log exists

Log files:  Blitz-0.log                    253.0 MB      errors=47   warnings=328
Log files:  Blitz-0.log.1                  524.0 MB      errors=72   warnings=1654
Log files:  Blitz-0.log.2                  524.0 MB      errors=109  warnings=121
Log files:  Blitz-0.log.3                  524.0 MB      errors=176  warnings=1280
Log files:  Blitz-0.log.4                  524.0 MB      errors=16   warnings=6552
Log files:  Blitz-0.log.5                  524.0 MB      errors=43   warnings=2886
Log files:  Blitz-0.log.6                  524.0 MB      errors=18   warnings=1360
Log files:  Blitz-0.log.7                  524.0 MB      errors=0    warnings=268
Log files:  Blitz-0.log.8                  524.0 MB      errors=0    warnings=474
Log files:  Blitz-0.log.9                  524.0 MB      errors=0    warnings=726
Log files:  DropBox.log                    8.0 KB
Log files:  FileServer.log                 0.0 KB
Log files:  Indexer-0.log                  311.0 MB      errors=246287 warnings=20936
Log files:  MonitorServer.log              7.0 KB
Log files:  OMEROweb.log                   161.0 KB      errors=208  warnings=0
Log files:  OMEROweb.log.1                 25.0 KB       errors=19   warnings=1
Log files:  OMEROweb.log.2                 35.0 KB       errors=30   warnings=1
Log files:  OMEROweb.log.3                 6.0 KB        errors=0    warnings=1
Log files:  OMEROweb.log.4                 50.0 KB       errors=12   warnings=0
Log files:  OMEROweb.log.5                 49.0 KB       errors=16   warnings=1
Log files:  OMEROweb_request.log           37.0 KB       errors=89   warnings=9
Log files:  PixelData-0.log                19.0 MB       errors=684  warnings=188
Log files:  Processor-0.log                1.0 MB        errors=8    warnings=4881
Log files:  Tables-0.log                   6.0 KB
Log files:  TestDropBox.log                n/a
Log files:  error500-2018-07-11 14:53:27.504951.html 2.0 KB        errors=3    warnings=0
Log files:  error500-2021-03-19 22:06:29.487246.html 2.0 KB        errors=3    warnings=0
Log files:  error500-2021-03-19 22:07:41.874412.html 2.0 KB        errors=3    warnings=0
Log files:  error500-2021-03-19 22:09:20.880147.html 2.0 KB        errors=3    warnings=0
Log files:  master.err                     0.0 KB        errors=1    warnings=1
Log files:  master.out                     161.0 KB      errors=12   warnings=0
Log files:  Total size                     5303.90 MB


Environment:OMERO_HOME=(unset)
Environment:OMERO_NODE=(unset)
Environment:OMERO_MASTER=(unset)
Environment:OMERO_USERDIR=(unset)
Environment:OMERO_TMPDIR=(unset)
Environment:PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/omero/bin
Environment:PYTHONPATH=(unset)
Environment:ICE_HOME=(unset)
Environment:LD_LIBRARY_PATH=(unset)
Environment:DYLD_LIBRARY_PATH=(unset)

Cannot list deployed applications.
OMERO data dir:'/OMERO'                       Exists? True      Is writable? False
OMERO temp dir:'/home/omero/omero/tmp'        Exists? True      Is writable? True   (Size: 0)

JVM settings: Blitz                         -Xmx1238m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Indexer                       -Xmx825m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Pixeldata                     -Xmx1238m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
JVM settings: Repository                    -Xmx825m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

OMERO.web status... [RUNNING] (PID 1975)
[omero@omerobox bin]$
[omero@omerobox bin]$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0      0 localhost:lorica-in         *:*                         LISTEN
tcp        0      0 *:http                      *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 localhost:ipp               *:*                         LISTEN
tcp        0      0 localhost:postgres          *:*                         LISTEN
tcp        0      0 localhost:smtp              *:*                         LISTEN
tcp        0      0 *:45086                     *:*                         LISTEN
tcp        0      0 172.16.0.33:ssh             172.16.0.69:60066           ESTABLISHED
tcp        0      0 172.16.0.33:ssh             172.16.0.69:60685           ESTABLISHED
tcp        0    144 172.16.0.33:ssh             172.16.0.69:59877           ESTABLISHED
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 localhost:ipp               *:*                         LISTEN
tcp        0      0 localhost:postgres          *:*                         LISTEN
tcp        0      0 localhost:smtp              *:*                         LISTEN
tcp        0      0 *:42561                     *:*                         LISTEN
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*
udp        0      0 *:922                       *:*
udp        0      0 *:33320                     *:*
udp        0      0 *:941                       *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:922                       *:*
udp        0      0 localhost:58833             localhost:58833             ESTABLISHED
udp        0      0 *:50401                     *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     12792  /tmp/.s.PGSQL.5432
unix  2      [ ACC ]     STREAM     LISTENING     12262  /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     8590   @/com/ubuntu/upstart
unix  16     [ ]         DGRAM                    11583  /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     12171  /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     11781  /var/run/rpcbind.sock
unix  2      [ ACC ]     STREAM     LISTENING     12092  /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     12669  /var/run/mcelog-client
unix  2      [ ]         DGRAM                    9152   @/org/kernel/udev/udevd
unix  2      [ ACC ]     STREAM     LISTENING     13019  public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     13026  private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     13030  private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     13034  private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     13038  private/defer
unix  2      [ ACC ]     STREAM     LISTENING     13042  private/trace
unix  2      [ ACC ]     STREAM     LISTENING     13046  private/verify
unix  2      [ ACC ]     STREAM     LISTENING     13050  public/flush
unix  2      [ ACC ]     STREAM     LISTENING     13054  private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     13058  private/proxywrite
unix  2      [ ACC ]     STREAM     LISTENING     13062  private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     13066  private/relay
unix  2      [ ACC ]     STREAM     LISTENING     13070  public/showq
unix  2      [ ACC ]     STREAM     LISTENING     13074  private/error
unix  2      [ ACC ]     STREAM     LISTENING     13078  private/retry
unix  2      [ ACC ]     STREAM     LISTENING     13082  private/discard
unix  2      [ ACC ]     STREAM     LISTENING     13086  private/local
unix  2      [ ACC ]     STREAM     LISTENING     13090  private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     13094  private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     13098  private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     13102  private/scache
unix  2      [ ]         DGRAM                    12331  @/org/freedesktop/hal/udev_event
unix  2      [ ACC ]     STREAM     LISTENING     12306  @/var/run/hald/dbus-cY7TQMMWjD
unix  2      [ ACC ]     STREAM     LISTENING     13183  /var/run/abrt/abrt.socket
unix  2      [ ACC ]     STREAM     LISTENING     12299  @/var/run/hald/dbus-y0HWiG6XYE
unix  2      [ ]         DGRAM                    804499
unix  2      [ ]         DGRAM                    804027
unix  2      [ ]         DGRAM                    803450
unix  3      [ ]         STREAM     CONNECTED     803420 /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     803419
unix  3      [ ]         STREAM     CONNECTED     802867
unix  3      [ ]         STREAM     CONNECTED     802866
unix  2      [ ]         DGRAM                    802863
unix  3      [ ]         STREAM     CONNECTED     801721
unix  3      [ ]         STREAM     CONNECTED     801720
unix  2      [ ]         DGRAM                    801717
unix  2      [ ]         DGRAM                    67167
unix  3      [ ]         STREAM     CONNECTED     14694  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14693
unix  3      [ ]         STREAM     CONNECTED     14668  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14667
unix  3      [ ]         STREAM     CONNECTED     14656  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14655
unix  3      [ ]         STREAM     CONNECTED     14565  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14564
unix  3      [ ]         STREAM     CONNECTED     14457
unix  3      [ ]         STREAM     CONNECTED     14456
unix  3      [ ]         STREAM     CONNECTED     14455
unix  3      [ ]         STREAM     CONNECTED     14454
unix  3      [ ]         STREAM     CONNECTED     14453
unix  3      [ ]         STREAM     CONNECTED     14452
unix  3      [ ]         STREAM     CONNECTED     14451
unix  3      [ ]         STREAM     CONNECTED     14450
unix  3      [ ]         STREAM     CONNECTED     14449
unix  3      [ ]         STREAM     CONNECTED     14448
unix  3      [ ]         STREAM     CONNECTED     14161  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     14160
unix  2      [ ]         DGRAM                    13247
unix  3      [ ]         STREAM     CONNECTED     13219
unix  3      [ ]         STREAM     CONNECTED     13218
unix  2      [ ]         DGRAM                    13185
unix  2      [ ]         DGRAM                    13135
unix  3      [ ]         STREAM     CONNECTED     13105
unix  3      [ ]         STREAM     CONNECTED     13104
unix  3      [ ]         STREAM     CONNECTED     13101
unix  3      [ ]         STREAM     CONNECTED     13100
unix  3      [ ]         STREAM     CONNECTED     13097
unix  3      [ ]         STREAM     CONNECTED     13096
unix  3      [ ]         STREAM     CONNECTED     13093
unix  3      [ ]         STREAM     CONNECTED     13092
unix  3      [ ]         STREAM     CONNECTED     13089
unix  3      [ ]         STREAM     CONNECTED     13088
unix  3      [ ]         STREAM     CONNECTED     13085
unix  3      [ ]         STREAM     CONNECTED     13084
unix  3      [ ]         STREAM     CONNECTED     13081
unix  3      [ ]         STREAM     CONNECTED     13080
unix  3      [ ]         STREAM     CONNECTED     13077
unix  3      [ ]         STREAM     CONNECTED     13076
unix  3      [ ]         STREAM     CONNECTED     13073
unix  3      [ ]         STREAM     CONNECTED     13072
unix  3      [ ]         STREAM     CONNECTED     13069
unix  3      [ ]         STREAM     CONNECTED     13068
unix  3      [ ]         STREAM     CONNECTED     13065
unix  3      [ ]         STREAM     CONNECTED     13064
unix  3      [ ]         STREAM     CONNECTED     13061
unix  3      [ ]         STREAM     CONNECTED     13060
unix  3      [ ]         STREAM     CONNECTED     13057
unix  3      [ ]         STREAM     CONNECTED     13056
unix  3      [ ]         STREAM     CONNECTED     13053
unix  3      [ ]         STREAM     CONNECTED     13052
unix  3      [ ]         STREAM     CONNECTED     13049
unix  3      [ ]         STREAM     CONNECTED     13048
unix  3      [ ]         STREAM     CONNECTED     13045
unix  3      [ ]         STREAM     CONNECTED     13044
unix  3      [ ]         STREAM     CONNECTED     13041
unix  3      [ ]         STREAM     CONNECTED     13040
unix  3      [ ]         STREAM     CONNECTED     13037
unix  3      [ ]         STREAM     CONNECTED     13036
unix  3      [ ]         STREAM     CONNECTED     13033
unix  3      [ ]         STREAM     CONNECTED     13032
unix  3      [ ]         STREAM     CONNECTED     13029
unix  3      [ ]         STREAM     CONNECTED     13028
unix  3      [ ]         STREAM     CONNECTED     13025
unix  3      [ ]         STREAM     CONNECTED     13024
unix  3      [ ]         STREAM     CONNECTED     13022
unix  3      [ ]         STREAM     CONNECTED     13021
unix  3      [ ]         STREAM     CONNECTED     13018
unix  3      [ ]         STREAM     CONNECTED     13017
unix  3      [ ]         STREAM     CONNECTED     13015
unix  3      [ ]         STREAM     CONNECTED     13014
unix  2      [ ]         DGRAM                    12971
unix  2      [ ]         DGRAM                    12666
unix  2      [ ]         DGRAM                    12631
unix  3      [ ]         STREAM     CONNECTED     12598  /var/run/acpid.socket
unix  3      [ ]         STREAM     CONNECTED     12597
unix  3      [ ]         STREAM     CONNECTED     12592  @/var/run/hald/dbus-y0HWiG6XYE
unix  3      [ ]         STREAM     CONNECTED     12591
unix  3      [ ]         STREAM     CONNECTED     12438  @/var/run/hald/dbus-y0HWiG6XYE
unix  3      [ ]         STREAM     CONNECTED     12436
unix  3      [ ]         STREAM     CONNECTED     12326  @/var/run/hald/dbus-cY7TQMMWjD
unix  3      [ ]         STREAM     CONNECTED     12325
unix  3      [ ]         STREAM     CONNECTED     12301  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     12300
unix  2      [ ]         DGRAM                    12266
unix  3      [ ]         STREAM     CONNECTED     12112  /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     12111
unix  3      [ ]         STREAM     CONNECTED     12107
unix  3      [ ]         STREAM     CONNECTED     12106
unix  2      [ ]         DGRAM                    11864
unix  3      [ ]         DGRAM                    9171
unix  3      [ ]         DGRAM                    9170


After reviewing the documentation for this version it looks like the issue may be with the fact that the server isn't listening 4064. 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1171/rpcbind
tcp        0      0 127.0.0.1:4080              0.0.0.0:*                   LISTEN      1975/python
tcp        0      0 0.0.0.0:80                  0.0.0.0:*                   LISTEN      1612/nginx
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1457/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1332/cupsd
tcp        0      0 127.0.0.1:5432              0.0.0.0:*                   LISTEN      1491/postmaster
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1577/master
tcp        0      0 0.0.0.0:45086               0.0.0.0:*                   LISTEN      1189/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      1171/rpcbind
tcp        0      0 :::22                       :::*                        LISTEN      1457/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1332/cupsd
tcp        0      0 ::1:5432                    :::*                        LISTEN      1491/postmaster
tcp        0      0 ::1:25                      :::*                        LISTEN      1577/master
tcp        0      0 :::42561                    :::*                        LISTEN      1189/rpc.statd
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               1171/rpcbind
udp        0      0 0.0.0.0:631                 0.0.0.0:*                               1332/cupsd
udp        0      0 0.0.0.0:922                 0.0.0.0:*                               1171/rpcbind
udp        0      0 0.0.0.0:33320               0.0.0.0:*                               1189/rpc.statd
udp        0      0 0.0.0.0:941                 0.0.0.0:*                               1189/rpc.statd
udp        0      0 :::111                      :::*                                    1171/rpcbind
udp        0      0 :::922                      :::*                                    1171/rpcbind
udp        0      0 :::50401                    :::*                                    1189/rpc.statd
[root@omerobox bin]# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 4064 -j ACCEPT
[root@omerobox bin]# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
[root@omerobox bin]# service iptables restart

no luck

Luckily we have sftp enabled so even so I thought maybe I could pull the image files and download them and maybe recreate a server with a more up to date os version and omero version but i cant seem to locate the repo of images.

Any insight would be helpful

Hi @Otto. Wellcome to image.sc!

:scream_cat: Wow! I feel obliged to give the standard warning:

If you cannot perform the upgrade at this time then we strongly recommend that you shut down your OMERO server until upgrade is possible. At the very least you should switch your server into read-only mode and block OMERO.blitz API access to all but omero-web. If you hold any private data on OMERO.server then you should firewall it so that login is available only to those who may read all the data

:+1:

Is there nothing in the /OMERO directory?

OMERO data dir:'/OMERO'                       Exists? True      Is writable? False

Would you have expected a different directory to be configured?

Do you see anything related to /OMERO in the log file?

All the best,
~Josh