Understand Tesseract's Limits for Handwritten Text OCR

What this is, in plain English

This entry explores the challenges of using Tesseract OCR (Optical Character Recognition) to extract text from handwritten documents. OCR is a technology that converts different types of documents, like scanned paper documents or images, into editable and searchable data. Tesseract is a popular, free, and open-source OCR engine.

The author of the source article tried to use Tesseract for a pile of handwritten notes, expecting it to be a quick task. However, they quickly discovered that while Tesseract is excellent for printed text, it struggles significantly with handwriting, especially when it’s messy, faded, or has notes crammed into margins. The author’s experience shows that getting good results for handwriting with Tesseract often requires extensive manual tuning and preprocessing for each document, making it unsuitable for automated workflows.

This entry explains why Tesseract faces these difficulties with handwriting and what the author attempted to improve accuracy. It highlights that for complex handwritten documents, a simple, reproducible, and highly accurate solution with Tesseract is not easily achievable for a beginner.

What you can use it for

• Understand OCR limitations: Learn why general-purpose OCR tools like Tesseract are not always suitable for complex tasks like handwritten text recognition.
• Digitize clear printed documents: Tesseract is highly effective for converting scanned images of printed text into digital text.
• Experiment with image preprocessing: See how techniques like contrast adjustment, sharpening, and binarization can affect OCR results, even if they don’t solve the core problem for handwriting.
• Explore open-source OCR: Understand the basic setup and usage of Tesseract and its Python wrapper, pytesseract, for simpler OCR tasks.

Tools you need

• Python (free): A programming language used to run the OCR scripts.
• Tesseract OCR (free): The open-source OCR engine that does the actual text recognition. You need to install this on your computer as a standalone program.
• Pillow (free): A Python library for working with images, used here for opening and manipulating document images.

How it actually works

The author’s journey shows that while Tesseract is powerful, it’s not a magic bullet for handwritten text. Here’s a summary of the approach and findings:

1. Install Python and Tesseract: First, you need Python installed on your computer. Then, you install Tesseract OCR as a separate program. The pytesseract library (which you install with pip) acts as a bridge, allowing Python to send images to Tesseract and receive the recognized text.

   # macOS or Linux
   pip install pytesseract Pillow numpy

   # Windows (PowerShell)
   pip install pytesseract Pillow numpy

2. Basic OCR attempt: The author started with a simple Python script to open an image and pass it directly to Tesseract for text extraction. This works well for clear, printed text but produced very poor results (e.g., “2 1/4 cups flour” became “2 114 cps flcar”) for handwritten documents.

   import pytesseract
   from PIL import Image
   from pathlib import Path
   
   def extract_document(image_path):
       img = Image.open(image_path)
       text = pytesseract.image_to_string(img)
       return text
   
   if __name__ == "__main__":
       # The author processed images from a folder named 'images'
       # You would replace 'images' with the path to your own image folder
       for image_file in sorted(Path("images").glob("*")):
           print(f"\n--- {image_file.name} ---")
           print(extract_document(image_file))

3. Image Preprocessing: To improve accuracy, the author then tried various image preprocessing techniques using the Pillow library. These included converting images to grayscale, adjusting contrast, sharpening, and binarization (converting to pure black and white). The goal was to make the handwriting clearer for Tesseract.

   import pytesseract
   from PIL import Image, ImageFilter, ImageEnhance, ImageOps
   import numpy as np
   from pathlib import Path
   
   def preprocess_document(image_path):
       img = Image.open(image_path)
       img = img.convert('L') # Convert to grayscale
       img = ImageOps.autocontrast(img, cutoff=2) # Auto-adjust contrast
       img = img.filter(ImageFilter.SHARPEN) # Sharpen image
       img = ImageEnhance.Contrast(img).enhance(1.8) # Boost contrast
       
       # Binarization (converting to black and white based on a threshold)
       img_array = np.array(img)
       threshold = np.mean(img_array)
       img_array = ((img_array > threshold) * 255).astype(np.uint8)
       return Image.fromarray(img_array)
   
   if __name__ == "__main__":
       # The author processed images from a folder named 'images'
       # You would replace 'images' with the path to your own image folder
       for image_file in sorted(Path("images").glob("*")):
           print(f"\n--- {image_file.name} ---")
           img = preprocess_document(image_file)
           print(pytesseract.image_to_string(img))

4. Results and Conclusion: While preprocessing helped some documents (e.g., boosting contrast for faded ink), it often hurt others (e.g., binarization destroyed subtle handwriting details). The author found that every fix was a trade-off, requiring manual tuning for each document. Ultimately, the accuracy for handwritten text remained low (30-40%), and the process was not automated or reproducible. The author concluded that Tesseract is great for printed text, but not for the complex handwritten documents they had.

Words you’ll see, explained

• OCR (Optical Character Recognition): Technology that converts images of text (like scanned documents) into machine-readable text data.
• Tesseract: A free and open-source OCR engine developed by Google, widely used for converting images of printed text into digital text.
• Pillow: A Python library that adds image processing capabilities to Python, allowing you to open, manipulate, and save many different image file formats.
• pytesseract: A Python wrapper for Tesseract, meaning it’s a library that lets you use Tesseract’s features directly from Python code.
• Preprocessing: Steps taken to clean up or enhance an image before feeding it to an OCR engine, such as adjusting brightness, contrast, or sharpness.
• Binarization: An image processing technique that converts a grayscale or color image into a black and white image, typically by setting pixels above a certain brightness threshold to white and those below to black.
• Contrast: The difference in brightness between the lightest and darkest areas of an image. Adjusting it can make text stand out more.
• Sharpening: An image processing technique that enhances the edges of objects in an image, making them appear clearer and more defined.

Original source

This entry is based on a blog post titled “The OCR Rabbit Hole” by awjudd, shared on the DEV Community platform. The author documented their experience and frustrations trying to use Tesseract for handwritten documents.

Notes & variations

• Do you even need this?: For handwritten documents, especially those with varied handwriting, faded ink, or complex layouts, Tesseract is generally not the best tool for high accuracy. Consider specialized handwriting recognition services (often paid, like Google Cloud Vision’s Handwriting OCR or Amazon Textract) or even human transcription services if high accuracy is critical. For clear, printed documents, Tesseract is an excellent and free option.
• Free-tier limits: Tesseract itself is entirely free and open-source, with no usage limits. The limitations come from its inherent capabilities, particularly with challenging inputs like handwriting, rather than a paid tier.
• Common pitfall: Expecting Tesseract to perform as well on handwritten text as it does on printed text. It’s optimized for consistent, machine-generated fonts, not the variability of human handwriting.
• Tip for better results (with printed text): Always start with the highest quality image possible. Ensure good lighting, minimal shadows, and a straight, clear scan of the document. For printed text, Tesseract’s performance is highly dependent on the input image quality.