ImageJ wiki is read-only

Hi. I am in the process of publishing a plugin, and planned on adding it to the update sites at sites.imagej.net. (this was previously done by creating a wiki-account, right?). Has the process for this changed now? What is the new procedure?

Hi @Anders_Lunde, thanks for raising this question! I made a new thread for these kind of requests.

3 Likes

Today, @hinerm and I finally fixed the failing deploys to maven.scijava.org. (The problem was caused by mod_evasive, an Apache module intended to stop DOS attacks. Unfortunately, it interacts badly with our Maven repository, because Maven sometimes wants to pull down many artifacts quickly from the repository, which mod_evasive misconstrues as a DOS.)

We also worked a lot on the fiji-builds job, which constructs the timestamped Fiji bundles and makes them available from downloads.imagej.net. We have continued to see SSH timeouts when connecting to the new server. I am in the process of reworking the script to facilitate faster testing and troubleshooting.

I know that many of you are highly inconvenienced by the imagej.net wiki’s continued downtime. E.g., several people have told us it is delaying paper submissions, because the manuscripts have links to resources on imagej.net. Quite a few people have pressed for a timeline. Here is my best estimate:

  • [Friday]
    • Finish fixing fiji-builds.
    • Attempt to reinstate a read-only version of the imagej.net wiki.
  • [Saturday]
    • Migrate the ImageJ update sites and wsr.imagej.net to new server.
    • Scan all JAR files to ensure checksums match expected values.
  • [Sunday]
  • [Next week]
    • Continue helping migrate imagej.net to GitHub Pages.
    • Continue working on all loose ends, of which there are many.

Meanwhile, the new imagej.net site continues to make good progress. Hopefully we will be able to replace the read-only wiki in 1-4 weeks. Please continue to bear with us as we move forward as quickly as we can.

14 Likes

Is this the reason that ij.init() does not work now I am trying to use pyimagej under anaconda?

@Fred_ckx Nope. That call relies on maven.scijava.org, which should be working. Can you either A) start a new topic on this forum with details of what goes wrong (please paste the error); or B) file an issue to the pyimagej repository with the same? Thanks!

You could also try ij = imagej.init('sc.fiji:fiji+net.imagej:imagej-legacy') to see if that works any better for you. We still need to cut new releases of net.imagej:imagej and sc.fiji:fiji incorporating many fixes, as well as new versions of imglyb, scyjava and pyimagej on PyPI & conda-forge. Once those releases are done (planned before end of August!), PyImageJ should have fewer potential setup woes.

Thanks. I’ll first try your suggestion. If my issue remains I’ll create a new topic.

1 Like

Today I fixed the fiji-builds CI job. It was a configuration issue with our hardware firewall preventing Travis CI from uploading to the downloads.imagej.net server in many cases. In the process of troubleshooting it, I ended up largely rewriting and improving the deployment script. (For those interested: it no longer uses bootstrapJ8.js but instead bootstraps Fiji using mvn scijava:populate-app.)

The script has also been split into multiple pieces, so if you ever want to build your own Fiji bundles from the command line, now you can:

git clone git://github.com/fiji/fiji-builds
./bootstrap-fiji.sh Fiji.app
./update-fiji.sh Fiji.app
./generate-archives.sh Fiji.app

I know, it’s a dream come true.

Now that the thorny server troubles are solved, tomorrow I will start trying to reinstate a read-only version of the imagej.net wiki.

5 Likes

i couldnt find it myself: is there a way to download DiameterJ plugin? Or (as i am using Fiji) is it included in some repository available through Fiji? Couldnt find an answer myself
through the webarchive i found on the wikisite the former link to be http://imagej.net/images/6/65/DiameterJ_Fiji-_1-018.zip - but no working alternative.

There is an update site https://sites.imagej.net/DiameterJ you could try turning on, which appears to serve all the needed things? I didn’t test it, but you could try. If it doesn’t work, we can post the binaries here for the moment to tide you over until the imagej.net site is back online.

I made good initial progress on a read-only version of imagej.net, but it’s not yet live. Hopefully I can finish it tomorrow.

As part of that work I made a full git clone of the MediaWiki history including all media, with the goal of merging it with the new Markdown-based ImageJ site’s history. The goal is to keep the edit history of all pages seamless across the boundary, even though we are changing platforms.

I also migrated 2 more sites to the new servers; 7 remain. 31/38=81%.

Here is my revised timeline for the next few days:

  • [Sunday]
    • Finish reinstating the imagej.net wiki as a static read-only site.
    • Clean up and share the MediaWiki git clone with the imagej.github.io dev team.
  • [Monday]
    • Migrate the ImageJ update sites and wsr.imagej.net to new server.
    • Scan all JAR files to ensure checksums match expected values.
  • [Tuesday]
    • Migrate remaining services from old server to new ones.
    • Wipe and reinstall OS on original web server box.
  • [Wednesday]
4 Likes

Hi. I am trying to add an update site and am told I need to create a ImageJ wiki account, but I cannot seem to create one since the site is down. Is there any way to get around this?

1 Like

Hey @kf62007,

the mechanism is deactivated at the moment for technical reasons. Please post in this thread and admins will take care of your request:

Cheers,
Robert

Today I made a full clone of the imagej.net MediaWiki history to git. Actually I did it twice, because the first time I forgot the User namespace, as @frauzufall kindly pointed out afterward. Then I cleaned up the commits to have the correct authors rather than MediaWiki codenames, and shared it with the developers of the new imagej.github.io site.

I also wrote a script that uses wget to extract static HTML content from the entire MediaWiki site. It’s running now; I will clean up the scraped site and publish it tomorrow.

4 Likes

Today I restored the main imagej.net website in read-only mode! :tada:

Please report problems here on this thread.

One issue I already noticed: pages with slashes are not yet available (e.g.: https://imagej.net/Fiji/Downloads). I will work on restoring those tomorrow.

For the git lovers among us:

  • The complete MediaWiki site history transformed into a git repository is available here.
  • The static version of the MediaWiki site is on GitHub here.
  • Both of these repositories are TEMPORARY as we transition. We will merge the MediaWiki site history with the new website repository soon.
11 Likes

The missing pages with slashes have been restored. I believe all wiki pages are now present. Please report back here if you come across anything missing.

There is a remaining issue where some embedded images say “Error creating thumbnail”. I will work on fixing those next.

5 Likes

ImageJ website migration

The ImageJ website migration team met today to solidify our timeline for phased migration to the new site. I have started a new thread with that info. I’d like to focus any discussion of the imagej.net website migration there, and use this thread here for more general server infrastructure discussion and progress reports.

Security configuration

@hinerm has been making progress on internal server config issues, especially tuning configuration for security packages and making it reproducible across all our machines.

Verifying integrity of JAR files on hosted update sites

I completed an initial full scan of all JARs on our hosted ImageJ update sites, in an effort to validate their continued integrity. Results posted here. While I don’t see any obvious concerns in the output, there are a lot of mismatched SHA1s. This is to be expected though, because some developers deploy one release build to Maven, while uploading a different build of the same release to the update site. In particular, this was the case for net.imagej:ij for a long time. Why? One reason is that for quite some time we used a ‘double-push-to-master’ workflow that could easily result in this situation, if the developer: A) edited the POM to a release version number; B) built it to make sure it builds; C) uploaded that locally built JAR artifact to their update site; D) pushed the changes to GitHub and waited for the CI to build and deploy it; and finally E) bumped to the next SNAPSHOT version, committed and pushed that to GitHub.

There are also some artifacts whose Maven GAVs couldn’t be determined, or were determined incorrectly. Most of that is due to uber-JARs with multiple META-INF/maven subfolders, or non-Maven-built JARs with no META-INF/maven info inside. In general it’s tough to know which Maven artifact (if any) a JAR file is in a vacuum.

Feel free to check it out and analyze further if you have time and interest. Suggestions are also welcome if you have ideas for additional steps we could take to further validate the artifacts.

Next steps

Tomorrow and over the weekend, my priorities are:

  1. Fix “Error creating thumbnail” errors on imagej.net pages.
  2. Migrate update.imagej.net, update.fiji.sc and sites.imagej.net to the new server. There will be some downtime, but I’ll try to flip the switch during off-peak hours.
  3. Migrate the last two remaining sites—mirror.imagej.net and wsr.imagej.net.

As always, will keep y’all posted on progress.

2 Likes

Today I migrated update.imagej.net, update.fiji.sc and sites.imagej.net to their new home. I tested upload capabilities from ImageJ, and it worked for me using my credentials, so hopefully all is well—but please let us know if you have any problems updating or uploading.

I also recovered and restored 76 missing media files to the read-only imagej.net site. I am fairly confident all media is served and available now—please let us know if you notice anything else missing.

Next, I will work on merging the MediaWiki history into the new imagej.github.io repository. Then I will migrate the last two sites (mirror.imagej.net and wsr.imagej.net), and fix the “Error creating thumbnails” errors on the read-only imagej.net site.

Hope you all have a nice weekend.

7 Likes

Yesterday, as I mentioned on the ImageJ website migration thread, I merged the MediaWiki edit history into the new website repository.

Today, I migrated the last few remaining services, notably wsr.imagej.net and mirror.imagej.net, from the old web server to their new homes. The old web server is now fully offline! :tada: The final steps will be backing up any necessary data from the machine, hardware maintenance and potential upgrades, then performing a full wipe and install of a new operating system.

There are still quite a few loose ends left to fix—for example:

  • Populate missing thumbnail images on static imagej.net pages.
  • Get the nightly sync of mirror.imagej.net with imagej.nih.gov/ij working again.
  • Set up rsync daemons to enable mirroring of ImageJ update sites and downloads.imagej.net.
  • Migrate archives from retired mailman lists to the new servers.
  • Hook up website regression tests to a CI system, so we know immediately if services stop working as intended.
  • Finish migration of openspim.org to GitHub Pages.

However, seeing as how we’ve reached a milestone with the old server no longer being in production, I’m going to close out this topic, as it has gotten very long. If you wish to continue monitoring development of the new imagej.net website, you can follow this thread:

And as always, if you encounter any issues with ImageJ web resources, please open a topic here on the forum in the Websites category describing your issue.

:beers: :revolving_hearts: :unicorn:

8 Likes