Vfrdtyky

I am kind of stuck with this problem, and I know there are so many questions about it on stack overflow but in my case. Nothing gives the expected result.

The Context:

Am using Android OpenCV along with Tesseract so I can read the MRZ area in the passport. When the camera is started I pass the input frame to an AsyncTask, the frame is processed, the MRZ area is extracted succesfully, I pass the extracted MRZ area to a function prepareForOCR(inputImage) that takes the MRZ area as gray Mat and Will output a bitmap with the thresholded image that I will pass to Tesseract.

The problem:

The problem is while thresholding the Image, I use adaptive thresholding with blockSize = 13 and C = 15, but the result given is not always the same depending on the lighting of the image and the conditions in general from which the frame is taken.

What I have tried:

First I am resizing the image to a specific size (871,108) so the input image is always the same and not dependant on which phone is used.
After resizing, I try with different BlockSize and C values

//toOcr contains the extracted MRZ area

Bitmap toOCRBitmap = Bitmap.createBitmap(bitmap);

Mat inputFrame = new Mat();

Mat toOcr = new Mat();

Utils.bitmapToMat(toOCRBitmap, inputFrame);

Imgproc.cvtColor(inputFrame, inputFrame, Imgproc.COLOR_BGR2GRAY);

TesseractResult lastResult = null;

for (int B = 11; B < 70; B++) {

    for (int C = 11; C < 70; C++){

        if (IsPrime(B) && IsPrime(C)){

            Imgproc.adaptiveThreshold(inputFrame, toOcr, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY, B ,C);

            Bitmap toOcrBitmap = OpenCVHelper.getBitmap(toOcr);

            TesseractResult result = TesseractInstance.extractFrame(toOcrBitmap, "ocrba");

            if (result.getMeanConfidence()> 70) {

                if (MrzParser.tryParse(result.getText())){

                    Log.d("Main2Activity", "Best result with " + B + " : " + C);

                    return result;

                }

            }

        }

    }

}

Using the code below, the thresholded result image is a black on white image which gives a confidence greater than 70, I can't really post the whole image for privacy reasons, but here's a clipped one and a dummy password one.

Using the MrzParser.tryParse function which adds checks for the character position and its validity within the MRZ, am able to correct some occurences like a name containing a 8 instead of B, and get a good result but it takes so much time, which is normal because am thresholding almost 255 images in the loop, adding to that the Tesseract call.

I already tried getting a list of C and B values which occurs the most but the results are different.

The question:

Is there a way to define a C and blocksize value so that it s always giving the same result, maybe adding more OpenCV calls so The input image like increasing contrast and so on, I searched the web for 2 weeks now I can't find a viable solution, this is the only one that is giving accurate results

You can use a clustering algorithm to cluster the pixels based on color. The characters are dark and there is a good contrast in the MRZ region, so a clustering method will most probably give you a good segmentation if you apply it to the MRZ region.

Here I demonstrate it with MRZ regions obtained from sample images that can be found on the internet.

I use color images, apply some smoothing, convert to Lab color space, then cluster the a, b channel data using kmeans (k=2). The code is in python but you can easily adapt it to java. Due to the randomized nature of the kmeans algorithm, the segmented characters will have label 0 or 1. You can easily sort it out by inspecting cluster centers. The cluster-center corresponding to characters should have a dark value in the color space you are using.
I just used the Lab color space here. You can use RGB, HSV or even GRAY and see which one is better for you.

After segmenting like this, I think you can even find good values for B and C of your adaptive-threshold using the properties of the stroke width of the characters (if you think the adaptive-threshold gives a better quality output).

import cv2

import numpy as np



im = cv2.imread('mrz1.png')

# convert to Lab

lab = cv2.cvtColor(cv2.GaussianBlur(im, (3, 3), 1), cv2.COLOR_BGR2Lab)



im32f = np.array(im[:, :, 1:3], dtype=np.float32)

k = 2 # 2 clusters

term_crit = (cv2.TERM_CRITERIA_EPS, 30, 0.1)

ret, labels, centers = cv2.kmeans(im32f.reshape([im.shape[0]*im.shape[1], -1]), 

                                  k, None, term_crit, 10, 0)

# segmented image

labels = labels.reshape([im.shape[0], im.shape[1]]) * 255

Some results: