Vfrdtyky

I have set up 3 Google Cloud Storge buckets and 3 functions (one for each bucket) that will trigger when a PDF file is uploaded to a bucket. Functions convert PDF to png image and do further processing.

When I am trying to create a 4th bucket and similar function, strangely it is not working. Even if I copy one of the existing 3 functions, it is still not working and I am getting this error:

Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 333, in run_background_function _function_handler.invoke_user_function(event_object) File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 199, in invoke_user_function return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions_v1beta2/worker.py", line 196, in call_user_function event_context.Context(**request_or_event.context)) File "/user_code/main.py", line 27, in pdf_to_img with Image(filename=tmp_pdf, resolution=300) as image: File "/env/local/lib/python3.7/site-packages/wand/image.py", line 2874, in __init__ self.read(filename=filename, resolution=resolution) File "/env/local/lib/python3.7/site-packages/wand/image.py", line 2952, in read self.raise_exception() File "/env/local/lib/python3.7/site-packages/wand/resource.py", line 222, in raise_exception raise e wand.exceptions.PolicyError: not authorized/tmp/tmphm3hiezy' @ error/constitute.c/ReadImage/412`

It is baffling me why same functions are working on existing buckets but not on new one.

UPDATE:
Even this is not working (getting "cache resources exhausted" error):

In requirements.txt:

google-cloud-storage

wand

In main.py:

import tempfile



from google.cloud import storage

from wand.image import Image



storage_client = storage.Client()



def pdf_to_img(data, context):

    file_data = data

    pdf = file_data['name']



    if pdf.startswith('v-'):

        return 



    bucket_name = file_data['bucket']



    blob = storage_client.bucket(bucket_name).get_blob(pdf)



    _, tmp_pdf = tempfile.mkstemp()

    _, tmp_png = tempfile.mkstemp()



    tmp_png = tmp_png+".png"



    blob.download_to_filename(tmp_pdf)

    with Image(filename=tmp_pdf) as image:

        image.save(filename=tmp_png)



    print("Image created")

    new_file_name = "v-"+pdf.split('.')[0]+".png"

    blob.bucket.blob(new_file_name).upload_from_filename(tmp_png)

Above code is supposed to just create a copy of image file which is uploaded to bucket.

This actually seems to be a show stopper for ImageMagick related functionalities using PDF format. Similar code deployed by us on Google App engine via custom docker is failing with the same error on missing authorizations.

I am not sure how to edit the policy.xml file on GAE or GCF but a line there has to be changed to:

<policy domain="coder" rights="read|write" pattern="PDF" />

@Dustin: Do you have a bug link where we can see the progress ?

Update:

I fixed it on my Google app engine container by adding a line in docker image. This directly changes the policy.xml file content after imagemagick gets installed.

RUN sed -i 's/rights="none"/rights="read|write"/g' /etc/ImageMagick-6/policy.xml

This is an upstream bug in Ubuntu, we are working on a workaround for App Engine and Cloud Functions.

While we wait for the issue to be resolved in Ubuntu, I followed @DustinIngram's suggestion and created a virtual machine in Compute Engine with an ImageMagick installation. The downside is that I now have a second API that my API in App Engine has to call, just to generate the images. Having said that, it's working fine for me. This is my setup:

Main API:

When a pdf file is uploaded to Cloud Storage, I call the following:

response = requests.post('http://xx.xxx.xxx.xxx:5000/makeimages', data=data)

Where data is a JSON string with the format {"file_name": file_name}

On the API that is running on the VM, the POST request gets processed as follows:

@app.route('/makeimages', methods=['POST'])

def pdf_to_jpg():

    file_name = request.form['file_name']



    blob = storage_client.bucket(bucket_name).get_blob(file_name)



    _, temp_local_filename = tempfile.mkstemp()

    temp_local_filename_jpeg = temp_local_filename + '.jpg'



    # Download file from bucket.

    blob.download_to_filename(temp_local_filename)

    print('Image ' + file_name + ' was downloaded to ' + temp_local_filename)



    with Image(filename=temp_local_filename, resolution=300) as img:

        pg_num = 0

        image_files = {}

        image_files['pages'] = 



        for img_page in img.sequence:

            img_page_2 = Image(image=img_page)

            img_page_2.format = 'jpeg'

            img_page_2.compression_quality = 70

            img_page_2.save(filename=temp_local_filename_jpeg)



            new_file_name = file_name.replace('.pdf', 'p') + str(pg_num) + '.jpg'

            new_blob = blob.bucket.blob(new_file_name)

            new_blob.upload_from_filename(temp_local_filename_jpeg)

            print('Page ' + str(pg_num) + ' was saved as ' + new_file_name)



            image_files['pages'].append({'page': pg_num, 'file_name': new_file_name})



            pg_num += 1



    try:

        os.remove(temp_local_filename)

    except (ValueError, PermissionError):

        print('Could not delete the temp file!')



    return jsonify(image_files)

This will download the pdf from Cloud Storage, create an image for each page, and save them back to cloud storage. The API will then return a JSON file with the list of image files created.

So, not the most elegant solution, but at least I don't need to convert the files manually.