Load image via GUI in Python

Hello, it’s a pleasure to work with your software! I was wondering if you have any resources on how I could read an image/file via the browser (probably JS backend?) and transfer it to any python script.
Further discussion can be found here: https://github.com/oeway/ImJoy/issues/205

Thanks, and all the best
Bene

Hi @beniroquai

Thanks for posting the question here. I think it will be useful for other people as well, so it great. We are currently working on improving the development documentation. Let me post the answer here again so more people can see this:

Basically, you need two plugins, a window plugin (e.g.: named pluginA) for the web interface and a native-python plugin (e.g.: named pluginB) for the computation part.

Let’s say you define a python function process_image in pluginB:

class ImJoyPlugin():
    ...
    def process_image(self, image):
        print(image)
        # process the image

In pluginA, you can call process_image with pluginB.process_image(image), for example, in a button click handler function:

function button_clicked(){
    const pluginB = api.getPlugin('pluginB')
    const img = ...
    pluginB.process_image(img)
}

Notice that, we are passing the img object from pluginA to pluginB across different language, not all the data types can be transferred. But any primitive data types or dictionary/array/list with only primitive data types can be send from js to python. For an image, the easiest way is to get a base64 encoded string of the image in javascript and decode in Python, vise versa.

You can take the Image Recognition plugin as a start which include the code to get an image and convert to base64 url.

You need to change these lines:

        readImageFile().then(()=>{
            this.predict();
        })

to

        readImageFile().then((base64_url)=>{
            const pluginB = api.getPlugin('pluginB')
            pluginB.process_image(base64_url)
        })

In pluginB, you need to then decode the base64 string into an actual image (ref):

from io import BytesIO
import base64
from PIL import Image
import numpy as np

class ImJoyPlugin():
    ...
    def process_image(self, base64_url):
        # remove encoding header
        data = base64_url.split('base64,')[1]
        # decode base64 to bytes
        img_bytes = base64.b64decode(data)
        buffer = BytesIO(img_bytes)
        # open the image as PIL image
        img = Image.open(buffer)

        arr = np.array(img)
        # process the image

If your python script produce another image which you want to display the result, you can define a function (e.g.: display_result()) in pluginA and call the function from pluginB. You will need to do encoding like this:

with open(output_image_file_path, 'rb') as f:
    data = f.read()
    result = base64.b64encode(data).decode('ascii')
    base64_url = 'data:image/png;base64,' + result
    # call pluginA.display_result(base64_url)
    # or, you can try:
    api.createWindow(type='imjoy/image', w=12, h=15, data={"src": base64_url})

In the future, we would like to provide a set of predefined types so you won’t need to do encoding yourself, for example, supporting those most commonly used python object (Image objet from PIL, figure from matplotlib, and also those object supported by IPython.display).

If you can give more details about your application, I may provide more specific suggestions.
Good luck!

1 Like