Payment of ₹ requested by HONEY VIG. Click the link to pay securely via Razorpay: Make 5000 payment Payment of ₹ requested by HONEY VIG. Click the link to pay securely via Razorpay: Make 5000 payment ⭐ If you would like to buy me a coffee, well thank you very much that is mega kind! : https://www.buymeacoffee.com/honeyvig
Hire a web Developer and Designer to upgrade and boost your online presence with cutting edge Technologies
Showing posts with label ocr. Show all posts
Showing posts with label ocr. Show all posts

Friday, October 21, 2022

Java OCR Web Project – Tesseract Optical Character Recoginition(OCR)

 This tutorial shows JAVA program which can be used to convert Image To Text pragmatically. First you have to install ‘tesseract-ocr-setup-3.02.02.exe’ in the location C:\Tesseract-OCR for this example.

  1. Project Structure in Eclipse
  2. You can update the location you installed the OCR here and the local host server information.
    Common_Things.java

    package common_things;
    public class Common_Things {
       public static String url="http://localhost:8080/OCR";
       public static String upload_path="C:\\Tesseract-OCR";
    }
    
  3. This class does the CMD prompt commands to generate out put file after processing OCR. The output file is generated is C:\Tesseract-OCR by default. You can change it if you want in this class.
    create_file.java

    package OCR;
    import java.io.PrintWriter;
    public class create_file {
    	public void create_output_file(String file_name) {
    		String[] command =
    	    {
    	        "cmd",
    	    };
    	    Process p;
    		try {
    	    p = Runtime.getRuntime().exec(command);
    	    new Thread(new SyncPipe(p.getErrorStream(), System.err)).start();
    	    new Thread(new SyncPipe(p.getInputStream(), System.out)).start();
    	    PrintWriter stdin = new PrintWriter(p.getOutputStream());
    	    stdin.println("cd..");
    	    stdin.println("cd..");
    	    stdin.println("cd..");
    	    stdin.println("cd..");
    	    stdin.println("c:");
    	    stdin.println("cd..");
    	    stdin.println("cd..");
    	    stdin.println("cd..");
    	    stdin.println("cd Tesseract-OCR ");
    	    stdin.println("tesseract "+file_name+" "+file_name);
    	    stdin.close();
    	    int returnCode = p.waitFor();
    	   // System.out.println("Return code = " + returnCode);
    		} catch (Exception e) {
    			// TODO Auto-generated catch block
    			e.printStackTrace();
    		}
    	}
    }	
    
  4. This class is for supporting CMD operations from JAVA program.
    SyncPipe.java

    package OCR;
    import java.io.InputStream;
    import java.io.OutputStream;
    class SyncPipe implements Runnable
    {
    public SyncPipe(InputStream istrm, OutputStream ostrm) {
          istrm_ = istrm;
          ostrm_ = ostrm;
      }
      public void run() {
          try
          {
              final byte[] buffer = new byte[1024];
              for (int length = 0; (length = istrm_.read(buffer)) != -1; )
              {
                  ostrm_.write(buffer, 0, length);
              }
          }
          catch (Exception e)
          {
              e.printStackTrace();
          }
      }
      private final OutputStream ostrm_;
      private final InputStream istrm_;
    }
    
  5. This class uploads the image to process to the location specified in Common_Things.java, For us the location is C:\Tesseract-OCR
    FileUploadHandler.java

    package Upload_Data;
    import java.io.File;
    import java.io.IOException;
    import java.io.PrintWriter;
    import java.sql.Connection;
    import java.sql.PreparedStatement;
    import java.util.Iterator;
    import java.util.List;
    import javax.servlet.http.HttpServlet;
    import javax.servlet.http.HttpServletRequest;
    import javax.servlet.http.HttpServletResponse;
    import org.apache.commons.fileupload.FileItem;
    import org.apache.commons.fileupload.FileItemFactory;
    import org.apache.commons.fileupload.disk.DiskFileItemFactory;
    import org.apache.commons.fileupload.servlet.ServletFileUpload;
    import common_things.Common_Things;
    public class FileUploadHandler extends HttpServlet {
    	private static final long serialVersionUID = 1L;
    	public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
    		doPost(request, response);
    	}
    	public void doPost(HttpServletRequest request, HttpServletResponse response) throws IOException {
    		Connection connection=null;
    	  	PreparedStatement ps=null;
    	  	String file_name=null;
    	  	String file_name_given=null;
    		response.setContentType("text/html");
    		PrintWriter out = response.getWriter();
    		boolean isMultipartContent = ServletFileUpload.isMultipartContent(request);
    		if (!isMultipartContent) {
    			return;
    		}
    		FileItemFactory factory = new DiskFileItemFactory();
    		ServletFileUpload upload = new ServletFileUpload(factory);
    		try {
    			List fields = upload.parseRequest(request);
    			Iterator it = fields.iterator();
    			if (!it.hasNext()) {
    				return;
    			}
    			while (it.hasNext()) {
    				FileItem fileItem = it.next();
    				boolean isFormField = fileItem.isFormField();
    				if (isFormField) {
    					 if(file_name==null){
    			    		   if(fileItem.getFieldName().equals("file_name")){
    			    			   file_name=fileItem.getString();
    			    		   } 
    			    	   }
    					 if(file_name_given==null){
    			    		   if(fileItem.getFieldName().equals("file_name_given")){
    			    			   file_name_given=fileItem.getString();
    			    		   } 
    			    	   }
    				} else {
    	    	    	  if(fileItem.getSize()>0){
    	    	    		  file_name=fileItem.getName();
    	    	    		  fileItem.write( new File(Common_Things.upload_path + File.separator + fileItem.getName()));
    	    	    	  }
    				}
    			}
    		} catch (Exception e) {
    			e.printStackTrace();
    		}finally{
    			if(connection!=null){
    				try {
    						connection.close();
    						System.out.println("connection closed");
    					}
    				 catch (Exception e2) {
    					e2.printStackTrace();
    				}
    			}
    			if(ps!=null){
    				try {
    					ps.close();
    					}
    				 catch (Exception e2) {
    					 e2.printStackTrace();
    				}
    			}
    			out.println("");
    			out.close();
    		}
    	}
    }
    
  6. web.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <web-app id="WebApp_ID" version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
    	<display-name>
    	OCR</display-name>
    	<welcome-file-list>
    	<welcome-file>index.html</welcome-file>
    		<welcome-file>index.htm</welcome-file>
    		<welcome-file>index.jsp</welcome-file>
    		<welcome-file>default.html</welcome-file>
    		<welcome-file>default.htm</welcome-file>
    		<welcome-file>default.jsp</welcome-file>
    	</welcome-file-list>
    	<servlet>
    		<servlet-name>FileUploadHandler</servlet-name>
    		<servlet-class>Upload_Data.FileUploadHandler</servlet-class>
    	</servlet>
    	<servlet-mapping>
    		<servlet-name>FileUploadHandler</servlet-name>
    		<url-pattern>/FileUploadHandler/*</url-pattern>
    	</servlet-mapping>
    </web-app>
    
  7. index.jsp
     	<%@ page language="java" contentType="text/html; charset=ISO-8859-1"
    pageEncoding="ISO-8859-1"%>
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
    <%@page import="common_things.Common_Things"%><html>
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
    <!-- meta -->
    <meta charset="utf-8">
    </head>
    <body>
          <form action="<%=Common_Things.url %>/FileUploadHandler/" enctype="multipart/form-data" method="post">
                Select<input type="file" name="file2" /><br>
                File name <input type="text" name="file_name_given"> <br><br>
                <input type="submit" value="upload" />
          </form>
    </body>
    </html>
    
  8. read.jsp
     	<%@ page language="java" contentType="text/html; charset=ISO-8859-1"
    pageEncoding="ISO-8859-1"%>
    <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
    <%@page import="common_things.Common_Things"%>
    <%@page import="OCR.create_file"%>
    <%@page import="OCR.read_file"%><html>
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
    <!-- meta -->
    <meta charset="utf-8">
    </head>
    <body>
         <%
         String file_name=(String)request.getParameter("file_name");
         String file_name_given=(String)request.getParameter("file_name_given");
         //creating the out put file after upload and reading
         create_file obj_create_file=new create_file();
         obj_create_file.create_output_file(file_name);
         //read the created out put file
         read_file obj_read_file=new read_file();
         String read=obj_read_file.read_file(file_name);
         %>
         <%=file_name %><br>
         <%=file_name_given %><br>
         <br>
         <br>
         <hr>
         <%=read %>
         <hr>
         <br><br>
         <br><br><br>
         read success!!...
    </body>
    </html>
    

Convert image to text using CMD Command Prompt, Tesseract Optical Character Recognition (OCR)

 

1.Download and Install the Tesseract ‘tesseract-ocr-setup-3.02.02.exe’.


Install in any location you want,

Eg. C:\OCR

2.After installing verify all the below files are available in the installed directory, we are going to use tesseract.exe for the operation.


3.Collect the image and save that we need to OCR to any location in the PC

Eg: C:\Image_to_OCR.jpg


4.Open Command Prompt CMD in Administrator mode,

5.The Command to execute the OCR is,

tesseract <imagename> <outputname>

Where <imagename> you want to give the path of image and <outputname> you want to give path of output text file. Note that the out put is always text file as from Tesseract Documentation.
So the command for our testing will be,

"C:\OCR\tesseract" "C:\Image_to_OCR.jpg" "C:\out"

6.Now run the command in CMD

7.Now it will create an output file in C: as out.txt

8.The possible error which I got on testing was Cannot Open input file. It was caused when I missed to give the image name ,

Tesseract Open Source OCR Engine v3.02 with Leptonica
Cannot open input file:
ie. “C:\OCR\tesseract” “C:\ ” “C:\out”
Or
“C:\OCR\tesseract” “C:\out”

Sunday, April 24, 2022

Evaluating the Robustness of OCR Systems

 evaluating the robustness of OCR systems (such as Tesseract or Google’s Cloud Vision) when adversarial samples are presented as inputs. It’s somewhere in-between fuzzing and adversarial samples crafting, on a black box, the main objective being the creation of OCR-proof images, with minimal amounts of noise.

It’s an old project that I recently presented at an International Security Summer School hosted by the University of Padua. I decided to also publish it here mainly because of the positive feedback received when presented at the summer school.

I’ll try to focus on methodology and results, which I consider being of interest, without diving into implementation details.

I published this ~1 year ago - not sure if it still works as described here. Hopefully it does, but I’m pretty sure Google made changes to the Vision engine since then.

Motivation

Let’s start with what I considered to be plausible use cases for this project and what problems it would be able to solve.

  • Confidentiality of text included in images? – It is no surprise to us that large services (that’s you, Google) will scan hosted images for texts in order to improve classification or extract user information. We might want some of that information to remain private.

  • Smart CAPTCHA? – This aims to improve the efficiency of CAPTCHAs by creating images which are easier to read by humans, thus reducing the discomfort, while also rendering OCR-based bots ineffective.

  • Defense against content generators? – This could serve as a defense mechanism against programs which scan documents and republish content (sometimes using different names) in order to gain undeserved merits.

Challenges

Now, let’s focus on the different constraints and challenges:

1. Complex / closed-source architecture

Modern OCR systems are more complex than basic convolutional neural networks as they need to perform multiple actions (e.g.: deskewing, layout detection, text rows segmentation), therefore finding ways to correctly compute gradients is a daunting task. Moreover, many of them do not provide access to source code thus making it difficult to use techniques such as FGSM or GANs.

2. Binarization

An OCR system usually applies a binarization procedure (e.g.: Otsu’s method) to the image before running it through the main classifier in order to separate the text from the background, the ideal output being pure black text on a clean white background.

This proves troublesome because it restricts the samples generator from altering pixels using small values: as an example, converting a black pixel to a grayish color will be reverted in the binarization process thus generating no feedback.

3. Adaptive classification

This is specific to Tesseract, which is rather deprecated nowadays - still very popular, though. Modern classifiers might be using this method, too. It consists of performing 2 iterations over the same input image. In the first pass, characters which can be recognized with a certain confidence are selected and used as temporary training data. In the second pass, the OCR attempts to classify characters which were not recognized in the first iteration, but using what it previously learned.

Considering this, having an adversarial generator which alters one character at a time might not work as expected since that character might appear later in the image.

4. Lower entropy

This refers to the fact that the input data is rather ‘limited’ for an OCR system when compared to… let’s say object recognition. As an example, images which contain 3D objects have larger variance than those which contain characters since the characters have a rather fixed shape and format. This should make it more difficult to create adversarial samples for character classifiers without applying distortions.

A direct consequence is that it greatly restricts the amount of noise that can be added to an image so that the readability is preserved.

5. Dictionaries

OCR systems will attempt to improve their accuracy by employing dictionaries with predefined words. Altering a single character in a word (i.e.: the incremental approach) might not be effective in this case.

Targeted OCR Systems

 

Tested locally on Tesseract 4.0 and remotely on Google’s Cloud Vision OCR

For this project, I used Tesseract 4.0 for prototyping and testing, as it had no timing restrictions and allowed me to run a fast, parallel model with high throughput so I could test if the implementation works as expected. Later, I moved to Google’s Cloud Vision OCR and tried some ‘remote’ fuzzing through the API.

Methodology

A rather simplified view of the flow; a feedback-based adversarial samples generator (in image: obfuscator) alters inputs in order to maximize the error of the OCR system

In order to be able to cover even black box cases, I used a genetic algorithm guided by the feedback of the targeted OCR system. We observe that the confidence of the classifier, alone, is not a good metric for this problem, a score function based on the Levenshtein distance and the amount of noise is employed.

One of the main problems here was the size of the search space which was partially solved by identifying regions of interest in the image and focusing only on these. Also, lots of parameter tuning…

Noise properties

Given the constraints, the following properties of the noise model must be matched:

  • high contrast – so it bypasses the binarization process and generates feedback
  • low density – in order to maintain readability by exploiting the natural low-filtering capability of the human vision

Applying salt-and-pepper noise in a smart manner will, hopefully, satisfy the constraints.

Working modes

Different working modes for small and large characters, in order to preserve readability. Both managed to entirely hide the given text when tested on Tesseract 4.0

Initially, the algorithm worked using only overtext mode, which applied noise in the rectangle which contained characters. However, this method is not the best choice for texts written using smaller characters mainly because there are less pixels that can be altered thus drastically lowering the readability even with minimal amounts of noise. For this special case, the decision to insert the noise in-between the text rows (artifacts) was taken in order to preserve the original characters. Both methods presented similar success rates in hiding texts from the targeted OCR system.

Just for fun, here’s what happens if the score function is inverted, which translates as “generate an image with as much noise as possible, but which can be read by OCR software”. Weird, but it’s still recognized…

Results on Tesseract

Promising results were achieved while testing against Tesseract 4.0. In the following figure is presented an early (non-final) sample in which the word “Random” is not recognized by Tesseract:

Tests on Google’s Cloud Vision Platform

This is where things get interesting.

The implemented score function can be maximized in 2 ways: hiding characters or tricking the OCR engine into adding characters which shouldn’t be there.

One of the samples managed to create a loop in the recognition process of Google’s Cloud Vision OCR, basically recognizing the same text multiple times. No DoS or anything (or I’m not aware of it), I’m still not sure if the loop persisted or not - it either produced a small number of iterations, failed (timed out?) or they had load balancers which compensated for this and used different instances.

Possible loop in the recognition process: the same text gets recognized multiple times. The bottom-left and the top-right corners are ‘merged’ into an oblique text row so the recognition process is sent back to already processed text.

Let’s take a closer look at the sample: below, you can see how the adversarial sample was interpreted by Google’s Cloud Vision OCR system. The image was submitted directly to the Cloud Vision platform via the “Try the

Rectangles returned by Cloud Vision indicate that additional text rows are ‘created’ during the recognition thus creating a loop

Also the ‘boring’ case where the characters ar

Once again, using the artifacts mode on a small text since larger texts are way easier to hide

Conclusions

It works, but the project reached its objective and is no longer in development. It seems difficult to create samples that work for all OCR systems (generalization).

Also, the samples are vulnerable to changes at the preprocessing stage in the OCR pipeline such as:

  • noise filtering (e.g.: median filters)
  • compression techniques (e.g.: Fourier compression)
  • downscaling->upscaling (e.g.: Autoencoders)

However, we can conclude that, using this approach, it is more challenging to mask small characters without making the text difficult to read. I compiled the following graph, in which are compared: the images generated by the algorithm (below 7% noise density) and a set of images that contain random noise (15% noise density). The 2 sets contain different images with characters of sizes: 12, 21, 36, 50. Each random noise set contains 62 samples for each size - average values were used.

Noise efficiency is computed by taking into account the Levenshtein distance and the total amount of noise in the image.

Interesting TODO’s

  • Extracting templates from samples and training a generator?
  • Exploiting directly the row segmentation feature?
  • Attacking Otsu’s binarization method?

Maybe someday…

Cite

Should you find this relevant to your work, you can cite the article using:


@inproceedings{sporici2018evaluation,
  title={An Evaluation of OCR Systems Against Adversarial Machine Learning},
  author={Sporici, Dan and Chiroiu, Mihai and Cioc{\^\i}rlan, Dan},
  booktitle={International Conference on Security for Information Technology and Communications},
  pages={126--141},
  year={2018},
  organization={Springer}
}

As characters get smaller, the efficiency of the noise added by the algorithm decreases - the random noise samples behave in an opposite manner.

API” option so, at the moment of testing, the results could be easily reproduced.

 

 

 

 

 

Improving Tesseract 4's OCR Accuracy through Image Preprocessing

 

In this work I took a look at Tesseract 4’s performance at recognizing characters from a challenging dataset and proposed a minimalistic convolution-based approach for input image preprocessing that can boost the character-level accuracy from 13.4% to 61.6% (+359% relative change), and the F1 score from 16.3% to 72.9% (+347% relative change) on the aforementioned dataset. The convolution kernels are determined using reinforcement learning; moreover, to simulate the lack of ground truth in realistic scenarios, the training set consists of only 30 images while the testing set includes 10,000.

The dataset in cause is called Brno Mobile, and contains colored photographs of typed text, taken with handheld devices. Factors such as blurriness, low resolution, contrast, brightness are contributing to making the images challenging for an OCR engine.

Resized image from the Brno dataset which contains text that was not recognized by Tesseract 4 during the evaluation (an empty string was returned)

During this experiment, the out of the box version of Tesseract 4 has been used, which implies:

  • no retraining of the OCR engine
  • no lexicon / dictionary augmentations
  • no hints about the language used in the dataset
  • no hints about segmentation methods; default (automatic) segmentation is used
  • default settings for the recognition engine (LSTM + Tesseract)

Problem Analysis

Tesseract 4 has proven great performance when tested on favorable datasets by achieving good balance between precision and recall. It is presumed that this evaluation is performed on images that resemble scanned documents or book pages (with or without additional preprocessing) in which the number of camera-caused distortions is minimal. Tests on the Brno dataset led to much worse performance that will be discussed later in the article.

In the above figure, a high precision indicates favorable True-Positives to False-Positives ratio thus revealing proper differentiation between characters (i.e. a relatively small number of misclassifications). Despite this, almost no improvements in recall can be observed when switching from the base classification method to the Long Short-Term Memory (LSTM) based Convolutional Recurrent Neural Network (CRNN) for sequence to sequence mapping.

“Despite being designed over 20 years ago, the current Tesseract classifier is incredibly difficult to beat with so-called modern methods.” - Ray Smith, author of Tesseract

I assume that further training for different fonts might not provide significant improvements and neither will a different model of classifier. Is there a chance that the classifier doesn’t receive the correct input?

It was pointed out in a previous article that Tesseract is not robust to noise; certain salt-and-pepper noise patterns disrupt the character recognition process, leading to large segments of text being completely ignored by the OCR engine - the infamous empty string. From empirical observations, these errors seem to occur either for a whole word or sentence or not at all thus suggesting a weakness in the segmentation methodology.

The existence of similar behavior, given images which present more natural distortions, is questioned - hence this experiment.

Black-box Considerations

Since analyzing Tesseract’s segmentation methods is a daunting task, I opted for an adaptive external image correction method. To avoid diving into Tesseract 4’s source code, the OCR engine is considered a black-box; in this case, an unsupervised learning method must be employed. This ensures easier transitions to other OCR engines as it doesn’t directly rely on concrete implementations but only on outputs - at the cost of processing power and optimality.

Proposed Solution

The solution consists in directly preprocessing images before they are fed to Tesseract 4. An adaptive preprocessing operation is required, in order to properly compensate for any image features that cause problems in the segmentation process. In other words, an input image must be adapted so it complies with Tesseract 4’s preferences and maximizes the chance of producing the correct output, preferably without performing down-sampling.

I choose a convolution-based approach for flexibility and speed; other articles tend to perform more rigid image adjustments (such as global changes in brightness, fixed-constant conversion to grayscale, histogram equalization, etc.). I preferred an approach that can properly learn to highlight or mask regions of the image according to various features. For this, the kernels are optimized using reinforcement learning using an actor-critic model. To be more specific, it relies on Twin Delayed Deep Deterministic Policy Gradient (TD3 for short), for discovering features which minimize the Levenshtein distance between the recognized text and the ground truth. I’ll not dive into implementation details of TD3 here as it would be somehow out of scope but think of it as a method of optimizing the following formula:

Where

is a kernel, and

is a tuple from the training set.

A short (simpler) proof of concept of the convolutional preprocessor is presented in this Google Colab. It uses a different architecture than the final one and has the purpose of verifying if the idea of using convolutions is feasible and offers good results. A comparison is presented between original and preprocessed images including recognized texts for each sample.

The final model is illustrated below, with ReLU activations after each convolution to capture nonlinearities and prevent having negative values as pixels’ colors.

To properly compensate for image coloring and reduce the number of channels (R, G, B), 1x1 convolutions are used. This prevents overfitting up to a point while also ensuring grayscale output. Further convolutions are applied only on the grayscale image.

Symmetry constraints are additionally enforced for each 3x3 kernel in order to minimize the number of trainable parameters and avoid overfitting. This means that for a 3x3 kernel only 6 variables out of 9 must be determined while the rest can be generated through mirroring. Below are the values I got for the five kernels (bold to emphasize symmetry):

#1 #2     #3    
0.7 0.2573 -0.3 0.3 0.3 -0.2996 0.3
1.3 0.3 1.3 -0.295 0.3 1.2949 0.3
1.3 0.2573 -0.3 0.3 -0.2802 0.2922 -0.2802

 

Comparison

I used 10,000 images from the testing set for the evaluation of the current methodology and compiled the following graphs. The differences between original and preprocessed samples are illustrated with three metrics of interest: Character Error Rate (CER), Word Error Rate (WER) and Longest Common Subsequence Error (LCSE). In this article, LCSE is computed as follows:

Additionally, I plotted everything in histogram format to properly see the distributions of errors. For CER and WER, it is easy to observe the spikes around 1 (100%) that suggest the aforementioned segmentation problem (at block-of-text level) produces the most frequent error (empty strings are returned so all characters are wrong). In certain situations, the WER is larger than 1 because the preprocessing step introduces artifacts near the border of the image thus leading to recognition of non-existent characters. When looking at the LCSE plot, a distribution shift can be seen from the original approximately gaussian shape with its peak (mode) near the average number of characters in an image (56.95) to a more favorable shape with overall lower error rates.

A numeric comparison is presented below:

Metric Original (Avg.) Preprocessed (Avg.)
CER 0.866 0.384
WER 0.903 0.593
LCSE 48.834 24.987
Precision 0.155 0.725
Recall 0.172 0.734
F1 Score 0.163 0.729

Takeaways

Significant improvements can be observed through this preprocessing operation. Moreover, the majority of errors probably do not occur in the sequence to sequence classifier (since all the recognized characters are erroneous and would contradict previous performance analysis). A page-segmentation issue when automatic mode is used seems more plausible. It is shown that an array of convolutions is sufficient, in this case, to decrease error rates substantially.

The OCR performance on the preprocessed images is overall better but not good enough to be reliable. A 38% character error rate is still a large setback. I’m pretty sure that better recognitions can be obtained with more fine-tuning, a more complex architecture for the convolutional preprocessor and a more diverse training set. However, the current implementation is already very slow to train which makes me question if the entire methodology is feasible from this point of view.

Cite

If you found this relevant to your work, you can cite the article using:


@article{sporici2020improving,
  title={Improving the Accuracy of Tesseract 4.0 OCR Engine Using Convolution-Based Preprocessing},
  author={Sporici, Dan and Cușnir, Elena and Boiangiu, Costin-Anton},
  journal={Symmetry},
  volume={12},
  number={5},
  pages={715},
  year={2020},
  publisher={Multidisciplinary Digital Publishing Institute}
}

 

#4     #5    
-0.2793 0.2395 0.2885 -0.294 -0.2905 -0.2939
0.2395 0.7119 0.3 0.3 1.162 -0.2905
0.28850.3-0.2828-0.23280.3-0.294

 

 

 

PyTorch CRNN: Seq2Seq Digits Recognition w/ CTC

 

This article discusses handwritten character recognition (OCR) in images using sequence-to-sequence (seq2seq) mapping performed by a Convolutional Recurrent Neural Network (CRNN) trained with Connectionist Temporal Classification (CTC) loss. The aforementioned approach is employed in multiple modern OCR engines for handwritten text (e.g., Google’s Keyboard App - convolutions are replaced with Bezier interpolations) or typed text (e.g., Tesseract 4’s CRNN Based Recognition Module).

For the sake of simplicity, the example I’ll be presenting performs only digit recognition but can be easily extended to accommodate more classes of characters.

The overall source code for this project is quite long so I’m providing a Google Colab document that includes a working example.

Previous Inadequacies and Justification

“Why not simply segment characters in the image and recognize them one by one?”

While the approach is, indeed, more straightforward and has been incorporated in older OCR engines, it has its caveats, especially when considering handwritten text. These are caused by the imperfections of the written characters which can produce segmentation issues thus attempting to recognize invalid glyphs or symbols.

Whereas the MNIST problem is considered solved thus implying that reliable classifiers can be constructed to individually recognize digits, the problem of correct segmentation still remains in realistic scenarios. Splitting or merging glyphs to form valid digits proves to be a difficult challenge and requires additional knowledge to be embedded into the segmentation module.

Seq2Seq Classifications

In this context, the main advantage brought by a seq2seq classifier is that it diminishes the impact of erroneous segmentations and takes advantage of the ability of a neural network to generalize. It only requires a valid segmentation of the word or text line in cause.

Consider the following simplistic model that has a sliding window or mask (no convolutions), of size (1, img_height). Each set of pixels covered by the sliding window is fed into a neural network made out of neurons with memory (e.g., GRU or LSTM); the job of the neural network is to take a sequence of such stripes and output recognized digits. 

Multiple digits will be included in a single sequence - because we’re feeding the network an image which contains more than a digit. It is up to the neural network to determine during the training phase how many stripes to take into account when classifying a digit (i.e., how much to memorize). The image below illustrates how a RNN should ‘group’ stripes together in order to recognize each digit in the sequence.

The RNN receives sequences of ‘vertical’ arrays of pixels (stripes) covered by the sliding window of width equal to 1; once trained, the RNN will be able to memorize that certain sequences of arrays (here in colors) form specific digits and properly separate multiple digits (i.e., ‘change the colors’) even though they are merged in the given image.

Using this method, it is possible to train a neural network by simply saying that the image above contains the numbers ‘55207’, without further information (e.g.: alignment, delimitations, bounding boxes etc.)

CTC and Duplicates Removal

CTC loss is most commonly employed to train seq2seq RNNs. It works by summing the probabilities for all possible alignments; the probability of an alignment is determined by multiplying the probabilities of having specific digits in certain slots. An alignment can be seen as a plausible sequence of recognized digits.

Going back to the ‘55207’ example, we can express the probability of the alignment

as follows:

To properly remove duplicates and also correctly handle numbers that contain repeating digits, the blank class is introduced, with the following rules:

  1. 2 (or more) repeating digits are collapsed into a single instance of that digit unless separated by blank - this compensates for the fact that the RNN performs a classification for each stripe that represents a part of a digit (thus producing duplicates)
  2. multiple consecutive blanks are collapsed into one blank - this compensates for the spacing before, after or between the digits

Given these aspects, there are multiple alignments that, once collapsed, lead to the correct alignment (‘55207’).

For example: 55-55222–07 once collapsed leads to ‘55207’ and suggests the correct sequence even though it has a different structure because of additional duplicates and blanks (marked as ‘-’ here). The probability of this alignment (

) is computed as previously shown but it also includes the probabilities of the blank class:

Finally, the CTC probability of a sequence is calculated, as previously mentioned, by summing the probabilities for all different alignments:

When training, the neural network attempts to maximize this probability for the sequence provided as ground truth.

A decoding method is used to recover the text from a set of digits probabilities; a naive approach would be to pick, for each slot in the alignment, the digits with the highest probability and then collapse the result. This approach is easier to implement and might be enough for this example although beam search (i.e.: greedy approach that picks first N digits with highest probabilities, instead of only one) is employed for such tasks in larger projects.

Including Convolutional Layers

Implementing convolutions in the previously described model simply implies that raw pixel information is replaced, in the input of the RNN, with higher level features. In PyTorch, the output of the convolution layers must be reshaped to the time sequence format (batch_size, sequence_length, gru_input_size).

In the current project, the output of the convolution part has the following shape: (batch_size, num_channels, convolved_img_height, convolved_img_width). I’m permuting the tensor to (batch_size, convolved_img_width, convolved_img_height, num_channels) and then reshaping the last 2 dimensions into one which becomes gru_input_size).

Dataset Generation

To avoid additional steps such as image preprocessing, segmentation and class balancing I picked a more friendly dataset: EMNIST for digits. The following helper script randomly picks digits from the dataset, applies affine augmentations and concatenates them into sequences of a given length.

CRNN Model

A LeNet-5 based convolution model is employed, with the following modifications:

  • 5x5 filters are replaced with 2 consecutive 3x3 filters
  • max-pooling is replaced with strided convolutions

The resulted higher level features are fed into a Bi-GRU RNN with a linear layer in the end which has 10 + 1 possible outputs ([0-9] digits + blank). I’ve chosen GRU over LSTM since it had similar results but required fewer resources. A log_softmax activation function is used in the final layer since it the loss function (PyTorch’s CTCLoss) requires a logarithmized version of the output; also, this should provide better numerical properties as it highly penalizes incorrect classifications.


class CRNN(nn.Module):

    def __init__(self):
        super(CRNN, self).__init__()

        self.num_classes = 10 + 1
        self.image_H = 28

        self.conv1 = nn.Conv2d(1, 32, kernel_size=(3,3))
        self.in1 = nn.InstanceNorm2d(32)

        self.conv2 = nn.Conv2d(32, 32, kernel_size=(3,3))
        self.in2 = nn.InstanceNorm2d(32)

        self.conv3 = nn.Conv2d(32, 32, kernel_size=(3,3), stride=2)
        self.in3 = nn.InstanceNorm2d(32)

        self.conv4 = nn.Conv2d(32, 64, kernel_size=(3,3))
        self.in4 = nn.InstanceNorm2d(64)

        self.conv5 = nn.Conv2d(64, 64, kernel_size=(3,3))
        self.in5 = nn.InstanceNorm2d(64)

        self.conv6 = nn.Conv2d(64, 64, kernel_size=(3,3), stride=2)
        self.in6 = nn.InstanceNorm2d(64)

        self.postconv_height = 3
        self.postconv_width = 31

        self.gru_input_size = self.postconv_height * 64
        self.gru_hidden_size = 128 
        self.gru_num_layers = 2
        self.gru_h = None
        self.gru_cell = None

        self.gru = nn.GRU(self.gru_input_size, self.gru_hidden_size, self.gru_num_layers, batch_first = True, bidirectional = True)

        self.fc = nn.Linear(self.gru_hidden_size * 2, self.num_classes)

    def forward(self, x):
        batch_size = x.shape[0]

        out = self.conv1(x) 
        out = F.leaky_relu(out)
        out = self.in1(out)

        out = self.conv2(out) 
        out = F.leaky_relu(out)
        out = self.in2(out)

        out = self.conv3(out)
        out = F.leaky_relu(out)
        out = self.in3(out)

        out = self.conv4(out)
        out = F.leaky_relu(out)
        out = self.in4(out)

        out = self.conv5(out)
        out = F.leaky_relu(out)
        out = self.in5(out)

        out = self.conv6(out)
        out = F.leaky_relu(out)
        out = self.in6(out)

        out = out.permute(0, 3, 2, 1) 
        out = out.reshape(batch_size, -1, self.gru_input_size)

        out, gru_h = self.gru(out, self.gru_h)
        self.gru_h = gru_h.detach()
        out = torch.stack([F.log_softmax(self.fc(out[i])) for i in range(out.shape[0])])

        return out

    def reset_hidden(self, batch_size):
        h = torch.zeros(self.gru_num_layers * 2, batch_size, self.gru_hidden_size)
        self.gru_h = Variable(h)

crnn = CRNN()
criterion = nn.CTCLoss(blank=10, reduction='mean', zero_infinity=True)
optimizer = torch.optim.Adam(crnn.parameters(), lr=0.001) 

When performing backpropagation, the CTCLoss method will take the following parameters:

  • log_probabilities - this is the output from the log_softmax
  • targets - a tensor which contains the expected sequence of digits
  • input_lengts - the length of the input sequence after it is processed by the convolutional layers (i.e. post-convolution width)
  • target_lengths - the length of the target sequence

The last 2 parameters (input_lengths and target_lengths) are used to instruct the CTCLoss function to ignore additional padding (in case you added padding to the imagine or the target sequences to fit them into a batch).

log_probabilities will look like a (T, C)-shaped tensor (T = number of timesteps, C = number of classes) and specifies, for teach timestep, the probability of it belonging in a specific class. This tensor is decoded into text using a best path (greedy) approach: for each timestep, this algorithm picks the class with the maximum probability while also collapsing multiple occurences of the same character into one (unless they’re separated by a blank).

In my implementation, I’ve used y_pred.permute(1, 0, 2) to reorder the CRNN’s output so it matches the CTCLoss’s desired input format.

Another aspect you should pay attention to is resetting the hidden state of the GRU layers (crnn.reset_hidden(batch_size)) before recognizing any new sequence; in my experience this provided better results.

Feel free to check the code on my Google colab (link above) for further details.

Results

I’ve tested the model using 10,000 generated sequences: 8,000 for training and 2,000 for testing. Below are the plots for training and testing loss and also the evolution of precision - I’m considering that the dataset is approximately balanced. A true positive (TP) is counted only when the recognized sequence entirely matches the ground truth. The results are not ideal but I think the current model represents a decent starting point for greater projects.

The CRNN manifests some overfitting behavior but the results are acceptable considering its purpose.

After 6 epochs, the CRNN successfully recognizes 7567 out of 8000 sequences in the training set and 1776 out of 2000 from the testing set.

References