Extracting positions of values within numeric arrays with a given threshold

Hi there,

I wonder if anyone knows of an easy way to extract data from a numeric array (using a threshold of say 0.3) and create a new array for each cluster of adjacent values?

Here’s an eg of the data set I work with:
a = newArray(0.328,0.177,0.3,0.241,0.634,0.604,0.602,0.826,0.75,0.765,0.59,0.623,0.544,0.635,0.948,1,0.918,
0.731,0.474,0.547,0.436,0.409,0.155,0.179,0.184,0.296,0.271,0.087);

Please advise.
Thanks.
Adam

Hello Adam -

If I understand what you are trying to do, I believe that:
Fiji / ImageJ doesn’t have any built-in tools or functions
that will do it out-of-the box; and it should be straightforward
to do it with one of the programming / scripting languages
supported by Fiji (e.g., java or jython (python)).

First, I think you want to build a set of arrays, each of which
is a cluster of adjacent values that meet or exceed your threshold.
So, using your example data and threshold, you would want to
build as output an array of arrays (or lists or whatever your
preferred data structure in your preferred language is):

   o = {
      { 0.328 },
      { 0.300 },
      { 0.634, 0.604, 0.602, 0.826, 0.750, 0.765, 0.590, 0.623, 0.544, 0.635, 0.948, 1.000, 0.918, 0.731, 0.474, 0.547, 0.436, 0.409 }
   };

Something like this should work:

  1. Create an empty output array, o. (This will be an array of arrays.)

  2. Create an empty temporary working array, w.

  3. Loop over your input array, a.

  4. For each value v in a, if
    v >= threshold append the value to w
    v < threshold and if w is not empty, append a copy of w to o
    as a new array element and empty w (so you can start building the
    next cluster)

If you want to do this inside of Fiji, you could package your
algorithm as a java plugin or jython script (or other supported
plugin or script). If you don’t need Fiji, you could write a
free-standing program or script (in a language of your choice).

Thanks, mm

Hi mm,

Thanks for getting back. That’s a good idea. I’ll give it a shot.

WJ

Hi everyone,

Here’s the code I built to address my initial question. It’s partially working and I’ve trouble finding out why this is the case. I’ll greatly appreciate if someone can point me in the right direction. Briefly, this code identifies the positions of values that are above a set threshold. In the eg array a, there’re segments of values that are above the set threshold. This code then generates the length of each segment and put that in array o. However, it seems that the code encounters problem generating the length of segments that are further down the input array and I’m having difficulty finding the cause of the problem.

a = newArray(0.01,0.02,0.3, 0.5,0.6,0.4,0.2,0.241,0.634,0.604,0,0.826,0.1,0.8);

threshold = 0.3;

w = newArray(); //array storing positions above threshold
o = newArray(); //array storing lengths of consecutive elements above threshold

segmentStart = newArray();

counter = 1;										//counter

for (i=0; i<a.length; i++) {
		if (a[i]>=threshold) {
			w = Array.concat(w, i);					//Generates array of positions above threshold
		}
	}

Array.show(w);

for (ii=0; ii+1<w.length; ii++) {

	if ((w[ii+1] - w[ii]) == 1) {					//check for consecutive elements. Adds 1 if it's consecutive
		counter + 1;
		counter++;
		}

	else if ((w[ii+1] - w[ii]) != 1 && counter > 1) {	//Appends prior count to array o once consecutive count has ended at a specific stretch
		o = Array.concat(o, counter);
		counter = 1;
		}

	else if ((w[ii+1] - w[ii]) != 1 && counter == 1) {
		o = Array.concat(o, counter);			//adds 1 to account for non-consecutive but above threshold count
		counter = 1;
		}
	
	}

Array.show(o);

Please kindly advise.
Thanks.
wj

Hello Adam (WJ?) -

It looks like the problem is that the final group of consecutive indices
in your array w never gets recorded in your array o. A group of
consecutive indices is terminated when either the next index is not
consecutive or when there is no next index, that is, when you have
come to the end of your array w. You don’t use this end-of-array
condition to add the length of the final set of consecutive indices to o.

Consider what happens – walk through your code line by line – when
a consists of a single value greater than your threshold:

a = newArray();
a = Array.concat (a, 0.5);

We will then have w = (0). Step through your loop:

for (ii=0; ii+1<w.length; ii++) {...}

and consider what this generates for the array o. Also consider the
case where a = (0.5, 0.5) and see what you get for o. This should
make clear the issue, and probably also how to fix it.

Some minor comments:

You can (and should) delete the line counter + 1; in this block:

if ((w[ii+1] - w[ii]) == 1) {   //check for consecutive elements. Adds 1 if it's consecutive
	// counter + 1;   this line does nothing, so you can leave it out
	counter++;
	}

Similarly, your should delete counter = 1; in this block because
counter is anyway guaranteed to be 1:

else if ((w[ii+1] - w[ii]) != 1 && counter == 1) {
	o = Array.concat(o, counter);   //adds 1 to account for non-consecutive but above threshold count
	// counter = 1;   // does nothing

Also, the two else if blocks should arguably be combined together.
This could give slightly cleaner code. (There’s no clear “best answer”
here – it’s largely a matter of taste.)

(It would be helpful when you post examples like this if you could also
tell us what result you got, as well as the result you expected.)

Thanks, mm

1 Like

Hi mm, thanks for getting back.

I understand the issues you kindly raised in my code. May I know what would you recommend to ensure that the final group in array w will be recorded in array o? You mentioned that the end-of array method is unsuitable and I wonder if you’ve any suggestions on how I should go about that. Would you recommend a wholesale change to my current adjacent element comparison strategy?

As for the eg you raised on:
Consider what happens – walk through your code line by line – when
a consists of a single value greater than your threshold:

a = newArray();
a = Array.concat (a, 0.5);

We will then have w = (0) . Step through your loop:

for (ii=0; ii+1<w.length; ii++) {...}

and consider what this generates for the array o . Also consider the
case where a = (0.5, 0.5) and see what you get for o . This should
make clear the issue, and probably also how to fix it.

I can write the following 2 if statements to address that:

if (w.length == 1) {							//single position that passes threshold
	o = Array.concat(o, counter);
}

if ((max - min + 1) == w.length) {							//single position that passes threshold
	o = Array.concat(o, length);
}

And thanks for your comments on code clarity.
wj

Hello WJ (Adam?) -

I might not have expressed myself clearly. When I said “You don’t
use this end-of-array condition …”, I didn’t mean that using this kind
of end-of-array condition would be unsuitable. I just meant than the
code you posted doesn’t have an end-of-array check to trigger
recording the final consecutive-index group. I think that using
some sort of and-of-array processing to record that final group
would make sense.

I think you can make your current approach work without any wholesale
change. I think that adding to your current approach some code that
captures the final group should be enough.

Just to be clear, I wasn’t suggesting that you add special processing
for the case that w.length == 1. Rather, you should think of this case
as a simple test case that your general algorithm should process
correctly.

If you have more questions (or if you just want to check after you
think you have things running correctly), feel free to post a complete,
runnable script that includes some simple test data, together with the
results it gives you, and what you think the correct result should be.

Thanks, mm

Hi mm,

Here’s what I currently have:

a = newArray(0.01,0.02,0.3, 0.5,0.6,0.4,0.2,0.241,0.634,0.604,0,0.826,0.1,0.8);

threshold = 0.3;

w = newArray(); //array storing positions above threshold
o = newArray(); //array storing lengths of consecutive elements above threshold

segmentStart = newArray();

counter = 1;										//counter

for (i=0; i<a.length; i++) {
		if (a[i]>=threshold) {
			w = Array.concat(w, i);					//Generates array of positions above threshold
		}
	}

Array.show(w);

for (ii=0; ii+1<w.length; ii++) {

	if ((w[ii+1] - w[ii]) == 1) {					//check for consecutive elements. Adds 1 if it's consecutive
		counter++;
		}

	else if ((w[ii+1] - w[ii]) != 1 && counter > 1) {	//Appends prior count to array o once consecutive count has ended at a specific stretch
		o = Array.concat(o, counter);
		counter = 1;
		}

	else if ((w[ii+1] - w[ii]) != 1 && counter == 1) {
		o = Array.concat(o, counter);			//adds 1 to account for non-consecutive but above threshold count
		}
	
	}

if ((w[w.length-1]-w[w.length-2]) == 1) {
	o[o.length] == o[o.length] +1;
	}
else {
	o = Array.concat(o, 1);
	}

Array.show(o);

This code outputs

  1. array w = (2, 3, 4, 5, 8, 9, 11, 13) //this array contains positions of elements in array a that are above threshold
  2. arrary o = (4, 2, 1, 1) //this array contains length of segments that consist of consecutive. i.e. 4: (2, 3, 4, 5); 2: (8, 9); 1: (11); 1: (13).

The above results are what I expected.

To address your point that the prior version misses out on the final group of consecutive indices, I wrote an if statement (outside of the for loop) to interrogate whether the last 2 elements are consecutive. This appends 1 to array o if it’s not consecutive. If it’s consecutive, it adds 1 to the last element in array o. Although this has been addressed, I suspect there’re still loopholes in the code as you’ll see next when I tried the code with another array, b.

As mentioned, this code works with array a. However, when I changed it to

b = newArray(0.01,0.02,0.3, 0.5,0.6,0.4,0.2,0.241,0.634,0.604,0,0.826,0.9,0.8); 

Result:
array w = (2, 3, 4, 5, 8, 9, 11, 12, 13) //this is right
array o = (4, 3) //this is wrong as I’m expecting:
array o = (4 ,2, 3)

I suspect the following could be causing the issues and I’m not sure how should I go about them:

  1. How do I re-initiate a count after consecutive elements have been identified in the earlier part of array w?
  2. And what would you write as an end-of-array check?
  3. With regards to my suggestion for special processing approach (for cases like w.length = 1), may I know how else would you’ve written it such that it’s more incorporated in the main loop instead?

Due to my limited knowledge in coding, I apologise for the incessant questions and I greatly appreciate your feedback.

Thanks mm.
wj

Hello WJ -

I’ve taken the ijm (ImageJ macro) code you posted and tweaked it a
little, but I have not changed the logic of your algorithm.

The main thing is that I organized the two input cases you posted,
together with some I made up, as a set of test cases, and then
wrapped your logic in a loop that runs through the test cases one
by one. Also, each test case has an expected result that is compared
with the result produced by the algorithm.

I also commented out or otherwise fixed a couple of lines of code
that would sometimes stop the processing (syntax errors and array
indices out of bounds).

(I also replaced the Array.show() statements with print statements
and added some indentation.)

Here is the complete, runnable ijm script:

// test cases -- note, ijm doesn't support arrays of arrays
test1 = newArray (0.01,0.02,0.3, 0.5,0.6,0.4,0.2,0.241,0.634,0.604,0,0.826,0.1,0.8);
result1 = newArray (4, 2, 1, 1);

test2 = newArray (0.01,0.02,0.3, 0.5,0.6,0.4,0.2,0.241,0.634,0.604,0,0.826,0.9,0.8);
result2 = newArray (4, 2, 3);

test3 = newArray();
result3 = newArray();

test4 = newArray (0.0, 0.0, 0.0);
result4 = newArray();

test5 = newArray (0.5, 0.5, 0.5);
result5 = newArray();
result5 = Array.concat (result5, 3);

test6 = newArray (0.5);
result6 = newArray();
result6 = Array.concat (result6, 1);

test7 = newArray (0.5, 0.0, 0.5, 0.5, 0.5);
result7 = newArray (1, 3);

nTest = 7

// a = newArray(0.01,0.02,0.3, 0.5,0.6,0.4,0.2,0.241,0.634,0.604,0,0.826,0.1,0.8);

for (test = 1; test <= nTest; test++) {
  if (test == 1) {
    a = test1;
    r = result1;
  }
  else if (test == 2) {
    a = test2;
    r = result2;
  }
  else if (test == 3) {
    a = test3;
    r = result3;
  }
  else if (test == 4) {
    a = test4;
    r = result4;
  }
  else if (test == 5) {
    a = test5;
    r = result5;
  }
  else if (test == 6) {
    a = test6;
    r = result6;
  }
  else if (test == 7) {
    a = test7;
    r = result7;
  }

  print ("test " + test + ":");

  print ("a = ...");
  Array.print (a);

  threshold = 0.3;

  w = newArray();  //array storing positions above threshold
  o = newArray();  //array storing lengths of consecutive elements above threshold

  segmentStart = newArray();

  counter = 1;   //counter

  for (i=0; i<a.length; i++) {
    if (a[i]>=threshold) {
      w = Array.concat(w, i);   //Generates array of positions above threshold
    }
  }

  // Array.show(w);
  print ("w = ...");
  Array.print (w);

  for (ii=0; ii+1<w.length; ii++) {

    if ((w[ii+1] - w[ii]) == 1) {   //check for consecutive elements. Adds 1 if it's consecutive
      counter++;
    }

    else if ((w[ii+1] - w[ii]) != 1 && counter > 1) {  //Appends prior count to array o once consecutive count has ended at a specific stretch
      o = Array.concat(o, counter);
      counter = 1;
    }

    else if ((w[ii+1] - w[ii]) != 1 && counter == 1) {
      o = Array.concat(o, counter);   //adds 1 to account for non-consecutive but above threshold count
    }

  }
  
  // if ((w[w.length-1]-w[w.length-2]) == 1) {  // fails when w is short
  endTest = (w.length >= 2);
  if (endTest)  endTest = ( (w[w.length-1]-w[w.length-2]) == 1 );
  if (endTest) {
    // o[o.length] == o[o.length] + 1;   // maybe you meant "=" instead of "=="?
  }
  else {
    o = Array.concat(o, 1);
  }

  // Array.show(o);
  print ("o = ...");
  Array.print (o);
  print ("r = ...");
  Array.print (r);

  print ("check o against expected result (r)");
  check = 1;
  if (o.length != r.length)  check = 0;
  else
    for (k = 0; k < r.length; k++)
      if (o[k] != r[k])  check = 0;

  if (check)  print ("test " + test + " PASSED");
  else        print ("test " + test + " FAILED");
  }

One minor nuisance – the separate variables for test1, test2,
etc., and the set of if (test == 1) ... set of if, else if
blocks – is due to the fact that ijm doesn’t support arrays of arrays.

You will see that this script, as you reported, gets your first test case
right, and your second test case wrong. It also gets some of my test
cases right and wrong.

I think that you are on track to have a correct, working algorithm,
but you need to analyze where it goes wrong on the failing test
cases, and adjust the core logic. Note, it’s (almost) always better
to fix the general logic so that it works for the general case, rather
than putting in a case-specific fix.

This business of setting up a set test cases that get run automatically
when you’re writing an algorithm is often a good approach. It helps
ensure that when you fix one thing you don’t break another. Also,
it makes sense to start with some very simple – even trivial – test
cases. Make sure that your logic works for those before moving on
to more complicated cases.

As for the various specific questions you asked, please see if you
can make things work for at least one or two of the simplest test
cases, and if you’re having trouble, please ask questions in the
context of those simple cases.

(As an aside, writing ijm scripts is very convenient if you are executing
a bunch of run() commands. But as you find that the logic you write
yourself (rather than that in the built-in ImageJ tools) becomes more
complicated, you might find it helpful to use a general-purpose scripting
language – such as python (jython) – for scripting, and java (the
language ImageJ is written in) for writing plugins.)

No problem. Please take it step by step, and feel free to post questions
as they come up.

Thanks, mm

1 Like

Hi mm,

Thanks for your suggestion on setting up test cases. I agree that fixing the script for specific scenarios is risky and I’ll try to generalise my script logic.

Really appreciate your help!
wj