Omero.matlab disconnecting upon synchronising events on server

Which version of Gradle are you using?

I have made some changes (not released yet) to work with Gradle 6. The best option is for now to build with gradle 5.2.
For that you can use gradlew i.e.

gradle wrapper --gradle-version=5.2.1
./gradlew build

Cheers

Jmarie

Thank you Jean-Marie. It was indeed the version of Gradle. Debian 10 ships a 4.x.x. I rather used sdkman to install the one you recommend + ice as documented on the server installation for Debian 10 in omero doc and it built fine.
Best
Jacques

Hello,
After testing the change of Ice.Trace.Network to 1 and the use of a modified omero-blitz.jar. However, it worked exactly as before, and I cannot find any debug message. Are they supposed to be in stdout and stderr or at some other place? Did you succeed in getting some debug verbosity? Did I miss something? Anyway, I am still stuck with this disconnecting issue. Below the details of the changes to do the test.
Thank you very much for your help
Best
Jacques

In further details, I edited src/main/java/omero/client.java

@@ -385,6 +385,7 @@ public class client {
         optionallySetProperty(id, "IceSSL.Ciphers", "(DH_anon.*AES)");
         optionallySetProperty(id, "IceSSL.VerifyDepthMax", "6");
         optionallySetProperty(id, "IceSSL.VerifyPeer", "0");
+        optionallySetProperty(id, "Ice.Trace.Network", "1");
         optionallySetProperty(id, "omero.block_size", Integer
             .toString(omero.constants.DEFAULTBLOCKSIZE.value));

and recompiled.

I also added the lines marked by + to the creation of the client using the loadOmero from the matlab client

    map=javaObject('java.util.HashMap');
    map.put('omero.user',username);
    map.put('omero.pass',password);
    map.put('omero.group',group);
    map.put('omero.host',server);
    map.put('omero.port',int16(4064));
+    map.put('--IceSSL.Trace.Security','1');
+    map.put('--IceSSL.Trace.Network','1');
loadOmero(map);

Hello Jean-Marie,

So the second attempt was the right one. I used the changes above and got some verbosity from ice but I am unsure of what it means. Here is the log (edited to remove part related to my code and irrelevant).
I also attached the m-code used here.
If I manage to reconnect, the issue comes from the fact that sometimes, matlab keeps disconnecting and reconnecting 1000 of times, forking each time and at some point reaching the limit of the system. I have processes with 5000-8000 threads…
Does the ice messages evoke something to you? Should I use other flags to get more?
Furthermore, I don’t know if the ice message are flushed at everyline, i.e. if the “time order” of the log is correct.

Thank you for your help.
Best

Jacques

Called omeroKeepAlive at 2021-04-01__12-25-39

Then some operations kept matlab busy (pseudo-hanged) until an other call to keep alive below (which failed)

Called omeroKeepAlive at 2021-04-01__13-06-48
Warning: calling omero_session.keepAlive; I met the exception:
id: #MATLAB:Java:GenericException
message: Java exception occurred:
	Ice.ConnectionLostException
	    error = 0
		at IceInternal.AsyncResultI.__wait(AsyncResultI.java:276)
		at omero.api.ServiceFactoryPrxHelper.end_keepAlive(ServiceFactoryPrxHelper.java:5772)
		at omero.api.ServiceFactoryPrxHelper.keepAlive(ServiceFactoryPrxHelper.java:5655)
		at omero.api.ServiceFactoryPrxHelper.keepAlive(ServiceFactoryPrxHelper.java:5642)
		at com.mathworks.jmi.NativeMatlab.SendMatlabMessage(Native Method)
		at com.mathworks.jmi.NativeMatlab.sendMatlabMessage(NativeMatlab.java:273)
		at com.mathworks.jmi.MatlabLooper.sendMatlabMessage(MatlabLooper.java:120)
		at com.mathworks.jmi.Matlab.mtFevalConsoleOutput(Matlab.java:1835)
		at com.mathworks.jmi.MatlabMCR.mtFevalConsoleOutput(MatlabMCR.java:1236)
		at com.mathworks.timer.MWCallback$callbackEvent.run(MWCallback.java:541)
stack: 	FILE : /home/jacques/svn_repository_fsfs2/Tracking_Matlab/Tracking_Matlab/rdir/OMERO.matlab/omeroKeepAlive.m FUNCTION : omeroKeepAlive/doKeepAlive LINE : 80
	FILE : /usr/local/MATLAB/R2020b/toolbox/matlab/iofun/@timer/timercb.m FUNCTION : timercb LINE : 102
	FILE : /usr/local/MATLAB/R2020b/toolbox/matlab/iofun/timercb.m FUNCTION : timercb LINE : 24
 
Stopping all omeroKeepAlive timers
Warning time stamp: 2021-04-01__13-06-48
Warning: Lost connection to secure session (and putatively the non secure session) 
> In warning_perso (line 32)
In omeroAttemptReconnect (line 27)
In omeroKeepAlive/doKeepAlive (line 91)
In timer/timercb (line 102)
In timercb (line 24)
In sliding_analysis (line 83)
In meta_analysis (line 392)
In semibatch_partial1234_bis (line 116)
In xml_job_read_Omero (line 148) 
Warning time stamp: 2021-04-01__13-06-48
Warning: Omero lost connection - (previous omeroKeepAlive call: 2021-04-01__11-33-47)
Total Mem used: 2.4707 GB, Max Mem for this java: 14.9336 GB, Free Mem: 0.981938 GB, i.e. 6.58%
Total number of garbage collections: 1141   Total time spent collecting (s): 95.187
(re)try to establish connection in 1 minute - Attempting reconnecting # 1
 
> In warning_perso (line 32)
In omeroAttemptReconnect (line 31)
In omeroKeepAlive/doKeepAlive (line 91)
In timer/timercb (line 102)
In timercb (line 24)
In sliding_analysis (line 83)
In meta_analysis (line 392)
In semibatch_partial1234_bis (line 116)
In xml_job_read_Omero (line 148) 
-- 4/1/21 11:34:47:257 Network: established ssl connection
   local address = 129.20.159.236:52134
   remote address = 129.20.159.125:4064
-- 4/1/21 11:34:47:381 Network: established tcp connection
   local address = 129.20.159.236:50092
   remote address = 129.20.159.125:4063
-- 4/1/21 11:37:06:978 Network: closed tcp connection
   local address = 129.20.159.236:50052
   remote address = 129.20.159.125:4063
   Ice.ConnectionLostException
       error = 0
-- 4/1/21 12:37:06:978 Network: closed tcp connection
   local address = 129.20.159.236:50092
   remote address = 129.20.159.125:4063
   Ice.ConnectionLostException
       error = 0
-- 4/1/21 12:38:24:858 Network: closed ssl connection
   local address = 129.20.159.236:52134
   remote address = 129.20.159.125:4064
   Ice.ConnectionLostException
       error = 0
-- 4/1/21 13:06:48:339 Network: established ssl connection
   local address = 129.20.159.236:52276
   remote address = 129.20.159.125:4064
-- 4/1/21 13:06:48:341 Network: closed ssl connection
   local address = 129.20.159.236:52276
   remote address = 129.20.159.125:4064
   Ice.ConnectionLostException
       error = 0

We then clear all java object related to omero to let java do the garbage collection and then close omero connection (client.closeSession();), see omero_close_connection.m

close omero connections

And then I recreate the connection using memorized credentials.

2021-04-01__13-07-48 : info : Created omero client: omero.client@46df7d9b and session session-8c261b68-50ef-458c-baa0-1b95239b91b1/d6782e27-e14d-4789-9faf-41851c065a5a -t -e 1.1 @ BlitzAdapters

2021-04-01__13-07-48 : info : Retrieved non secured omero session: omero.client@67b2b30a and client session-4e2c0dfc-0b15-46f2-96ce-2f4d891a3f9f/d6782e27-e14d-4789-9faf-41851c065a5a -t -e 1.1 @ BlitzAdapters

2021-04-01__13-07-48 : info : Connection restored

The m-codes forgotten above.
rdir.zip (32.3 KB)
And the full log for the sake of completeness.
full.log.zip (1.0 MB)
Thank you,
Best

Hi Jacques
Thanks for sharing the code, we will have to run more tests.
I am wondering if the combination Matlab&Java is not the source of the problem. Do you see similar problems if you use directly Java?
Did you start seeing the problem after upgrading Matlab?

Nothing obvious comes to mind leading to the porblem

Cheers
Jmarie

Hi Jean-Marie,
thank you for the tests. I am looking forward the results. The issue seems not to have started with a matlab upgrade. Maybe with a minor update (within 2020b) or by upgrading linux (Debian 10). I currently use 2020b but had not the issue in the early time with this same version in October, or with 2020a. In both case, I used the same java 8 version 192-b12. Indeed, I was not able to use the matlab internal version because of Matlab 2019b cannot connect to omero.
Is it useful if test something ? Are you expecting a change with 2021a… with the same java version or with the embedded version ? In the second case, I still need to fix the issue mentioned in my other post (URL above).
Thank you
Best
Jacques

Hi Jacques

I did not have time to test yet due to other deadlines.
The Java issue is still concerning: I have looked at other Java version (not “combined” with Matlab) and do not have any problems.
I will keep you posted

Cheers

Jmarie

Hi Jean-Marie,

I uptated finally to Matlab 2021a and consequently switched to Java 8 b202. The certificates in omero made the trick. I am currently testing and will let you know the outcome.
Best
Jacques