Building Desktop Passport Scanner with Qt and USB Camera

Xiao Ling - Sep 6 '21 - - Dev Community

If you search for passport scanner software or MRZ reader software, you will find many of them are only available for mobile devices. For police officers, scanning passports by mobile devices is convenient when they are patrolling. However, for customs and immigration officers, they usually use desktop system and professional passport scanner or reader, which cost a lot, to check passengers' passport information. Dynamsoft's OCR SDK is available for both mobile and desktop scenarios. In this article, I will demonstrate an economic way that uses a cheap USB web camera (less than $20), Qt, and Dynamsoft MRZ SDK to build a desktop passport scanner application for Windows and Linux.

Prerequisites

The Skeleton of Qt C++ Project for Desktop Passport Scanner

Before getting started, let's get the codebase of the barcode scanning application that I implemented recently.

git clone https://github.com/yushulx/Qt-desktop-barcode-reader.git
Enter fullscreen mode Exit fullscreen mode

The codebase has implemented the file loading and camera video streaming functions. What I need to do is to replace barcode recognition SDK with MRZ recognition SDK. In addition, the project needs to import extra character models (trained by Caffe) for OCR and a template file for providing MRZ recognition parameters.

Character model files

NumberUppercase.caffemodel
NumberUppercase.prototxt
NumberUppercase.txt
NumberUppercase_Assist_1lIJ.caffemodel
NumberUppercase_Assist_1lIJ.prototxt
NumberUppercase_Assist_1lIJ.txt
NumberUppercase_Assist_8B.caffemodel
NumberUppercase_Assist_8B.prototxt
NumberUppercase_Assist_8B.txt
NumberUppercase_Assist_8BHR.caffemodel
NumberUppercase_Assist_8BHR.prototxt
NumberUppercase_Assist_8BHR.txt
NumberUppercase_Assist_number.caffemodel
NumberUppercase_Assist_number.prototxt
NumberUppercase_Assist_number.txt
NumberUppercase_Assist_O0DQ.caffemodel
NumberUppercase_Assist_O0DQ.prototxt
NumberUppercase_Assist_O0DQ.txt
NumberUppercase_Assist_upcase.caffemodel
NumberUppercase_Assist_upcase.prototxt
NumberUppercase_Assist_upcase.txt
Enter fullscreen mode Exit fullscreen mode

Template file

{
   "CharacterModelArray" : [
    {
      "DirectoryPath": "CharacterModel",
      "FilterFilePath": "",
      "Name": "NumberUppercase"
    }
   ],
   "LabelRecognizerParameterArray" : [
      {
         "BinarizationModes" : [
            {
               "BlockSizeX" : 0,
               "BlockSizeY" : 0,
               "EnableFillBinaryVacancy" : 1,
               "LibraryFileName" : "",
               "LibraryParameters" : "",
               "Mode" : "BM_LOCAL_BLOCK",
               "ThreshValueCoefficient" : 15
            }
         ],
         "CharacterModelName" : "NumberUppercase",
         "LetterHeightRange" : [ 5, 1000, 1 ],
         "LineStringLengthRange" : [44, 44],
         "MaxLineCharacterSpacing" : 130,
         "LineStringRegExPattern" : "(P[OM<][A-Z]{3}([A-Z<]{0,35}[A-Z]{1,3}[(<<)][A-Z]{1,3}[A-Z<]{0,35}<{0,35}){(39)}){(44)}|([A-Z0-9<]{9}[0-9][A-Z]{3}[0-9]{2}[(01-12)][(01-31)][0-9][MF][0-9]{2}[(01-12)][(01-31)][0-9][A-Z0-9<]{14}[0-9<][0-9]){(44)}",
         "MaxThreadCount" : 4,
         "Name" : "locr",
         "TextureDetectionModes" :[
            {
                "Mode" : "TDM_GENERAL_WIDTH_CONCENTRATION",
                "Sensitivity" : 8
            }
         ],
         "ReferenceRegionNameArray" : [ "DRRegion" ]
      }
   ],
   "LineSpecificationArray" : [
    {
        "Name":"L0",
        "LineNumber":"",
        "BinarizationModes" : [
            {
               "BlockSizeX" : 30,
               "BlockSizeY" : 30,
               "Mode" : "BM_LOCAL_BLOCK"
            }
         ]
    }
    ],
   "ReferenceRegionArray" : [
      {
         "Localization" : {
            "FirstPoint" : [ 0, 0 ],
            "SecondPoint" : [ 100, 0 ],
            "ThirdPoint" : [ 100, 100 ],
            "FourthPoint" : [ 0, 100 ],
            "MeasuredByPercentage" : 1,
            "SourceType" : "LST_MANUAL_SPECIFICATION"
         },
         "Name" : "DRRegion",
         "TextAreaNameArray" : [ "DTArea" ]
      }
   ],
   "TextAreaArray" : [
      {
         "LineSpecificationNameArray" : ["L0"],
         "Name" : "DTArea",
         "FirstPoint" : [ 0, 0 ],
         "SecondPoint" : [ 100, 0 ],
         "ThirdPoint" : [ 100, 100 ],
         "FourthPoint" : [ 0, 100 ]
      }
   ]
}
Enter fullscreen mode Exit fullscreen mode

Desktop Passport Scanner for Windows and Linux

Since we have already got the codebase, it won't take too much time to get the application work.

Library Linking

We extract the library files from the downloaded archive and put them into the corresponding folders:

  • Windows

    Copy DynamsoftLabelRecognizerx64.lib to platform/windows/lib.

    Copy DynamicPdfx64.dll, DynamsoftLabelRecognizerx64.dll, DynamsoftLicenseClientx64.dll and vcomp140.dll to platform/windows/bin.

  • Linux

    Copy libDynamicPdf.so, libDynamsoftLabelRecognizer.so, and libDynamsoftLicenseClient.so to platform/linux.

After that, we update the CMakeLists.txt file to link libraries and copy model and template files:

if (CMAKE_HOST_WIN32)
    target_link_libraries(${PROJECT_NAME} PRIVATE Qt5::Widgets Qt5::MultimediaWidgets "DynamsoftLabelRecognizerx64")    
elseif(CMAKE_HOST_UNIX)
    target_link_libraries(${PROJECT_NAME} PRIVATE Qt5::Widgets Qt5::MultimediaWidgets "DynamsoftLabelRecognizer")
endif()

# Copy DLLs
if(CMAKE_HOST_WIN32)
    add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD 
        COMMAND ${CMAKE_COMMAND} -E copy_directory
        "${PROJECT_SOURCE_DIR}/platform/windows/bin/"      
        $<TARGET_FILE_DIR:${PROJECT_NAME}>)
endif()

# Copy template
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD 
        COMMAND ${CMAKE_COMMAND} -E copy_directory
        "${PROJECT_SOURCE_DIR}/template/"      
        $<TARGET_FILE_DIR:${PROJECT_NAME}>)

# Copy model files
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD 
        COMMAND ${CMAKE_COMMAND} -E copy_directory
        "${PROJECT_SOURCE_DIR}/CharacterModel"      
        $<TARGET_FILE_DIR:${PROJECT_NAME}>/CharacterModel)
Enter fullscreen mode Exit fullscreen mode

Steps to Modify the Code for MRZ Recognition

Next, we import DynamsoftLabelRecognizer.h and DynamsoftCore.h to mainwindow.h:

#include "DynamsoftLabelRecognizer.h"
#include "DynamsoftCore.h"
Enter fullscreen mode Exit fullscreen mode

In mainwindow.cpp, we search for the line of invoking DBR_DecodeFile() method and then replace the line with:

int errorCode = DBR_DecodeFile(reader, fileName.toStdString().c_str(), "");
Enter fullscreen mode Exit fullscreen mode

Add the following code block to get passport information:

DLR_Result **results = handler->results;
for (int ri = 0; ri < handler->resultsCount; ++ri)
{
    DLR_Result* result = handler->results[ri];
    int lCount = result->lineResultsCount;
    for (int li = 0; li < lCount; ++li)
    {
        DM_Point *points = result->lineResults[li]->location.points;
        int x1 = points[0].x, y1 = points[0].y;
        int x2 = points[1].x, y2 = points[1].y;
        int x3 = points[2].x, y3 = points[2].y;
        int x4 = points[3].x, y4 = points[3].y;
    }

    if (lCount < 2)
    {
        continue;
    }
    string line1 = result->lineResults[0]->text;
    string line2 = result->lineResults[1]->text;
    if (line1.length() != 44 || line2.length() != 44)
    {
        continue;
    }
    if (line1[0] != 'P')
        continue;
    else {
        // Type
        string tmp = "Type: ";
        tmp.insert(tmp.length(), 1, line1[0]);
        out += QString::fromStdString(tmp) + "\n";

        // Issuing country
        tmp = "Issuing country: "; line1.substr(2, 5);
        tmp += line1.substr(2, 3);      
        out += QString::fromStdString(tmp) + "\n";

        // Surname
        int index = 5;
        tmp = "Surname: ";
        for (; index < 44; index++)
        {
            if (line1[index] != '<')
            {
                tmp.insert(tmp.length(), 1, line1[index]);
            }
            else 
            {
                break;
            }
        }
        out += QString::fromStdString(tmp) + "\n";

        // Given names
        tmp = "Given Names: ";
        index += 2;
        for (; index < 44; index++)
        {
            if (line1[index] != '<')
            {
                tmp.insert(tmp.length(), 1, line1[index]);
            }
            else 
            {
                tmp.insert(tmp.length(), 1, ' ');
            }
        }
        out += QString::fromStdString(tmp) + "\n";

        // Passport number
        tmp = "Passport number: ";
        index = 0;
        for (; index < 9; index++)
        {
            if (line2[index] != '<')
            {
                tmp.insert(tmp.length(), 1, line2[index]);
            }
            else 
            {
                break;
            }
        }
        out += QString::fromStdString(tmp) + "\n";

        // Nationality
        tmp = "Nationality: ";
        tmp += line2.substr(10, 3);
        out += QString::fromStdString(tmp) + "\n";

        // Date of birth
        tmp = line2.substr(13, 6);
        tmp.insert(2, "/");
        tmp.insert(5, "/");
        tmp = "Date of birth (YYMMDD): " + tmp;
        out += QString::fromStdString(tmp) + "\n";

        // Sex
        tmp = "Sex: ";
        tmp.insert(tmp.length(), 1, line2[20]);
        out += QString::fromStdString(tmp) + "\n";

        // Expiration date of passport
        tmp = line2.substr(21, 6);
        tmp.insert(2, "/");
        tmp.insert(5, "/");
        tmp = "Expiration date of passport (YYMMDD): " + tmp;
        out += QString::fromStdString(tmp) + "\n";

        // Personal number
        if (line2[28] != '<')
        {
            tmp = "Personal number: ";
            for (index = 28; index < 42; index++)
            {
                if (line2[index] != '<')
                {
                    tmp.insert(tmp.length(), 1, line2[index]);
                }
                else 
                {
                    break;
                }
            }
            out += QString::fromStdString(tmp) + "\n";
        }
    }
}
DLR_FreeResults(&handler);
Enter fullscreen mode Exit fullscreen mode

So far, the static image recognition is completed. In the following, we will implement real-time passport scanning by camera video stream.

To store MRZ recognition information and share it between threads, we create a new class MRZInfo:

#ifndef MRZINFO_H
#define MRZINFO_H

#include <QString>

class MRZInfo
{
public:
    MRZInfo() = default;

    ~MRZInfo(){};

    bool isNull();

public:
    QString text;
    int x1, y1, x2, y2, x3, y3, x4, y4, xx1, yy1, xx2, yy2, xx3, yy3, xx4, yy4;
};

#endif // MRZINFO_H
Enter fullscreen mode Exit fullscreen mode

Open work.h to add a new slot function detectMRZ(), which works in a worker thread for recognizing MRZ:

void Work::detectMRZ()
{
    while (m_bIsRunning)
    {
        QImage image;
        m_mutex.lock();
        // wait for QList
        if (queue.isEmpty())
        {
            m_listIsEmpty.wait(&m_mutex);
        }

        if (!queue.isEmpty())
        {
            image = queue.takeFirst();
        }
        m_mutex.unlock();

        if (!image.isNull())
        {
            // Detect MRZ
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

To recognize MRZ and extract passport information, we firstly convert QImage to ImageData and then call DLR_RecognizeByBuffer() method:

// Convert QImage to ImageData
ImageData data;
data.bytes = (unsigned char *)image.bits();
data.width = image.width();
data.height = image.height();
data.stride = image.bytesPerLine();
data.format = IPF_ARGB_8888;
data.bytesLength = image.byteCount();

QDateTime start = QDateTime::currentDateTime();
int errorCode = DLR_RecognizeByBuffer(recognizer, &data, "locr");
QDateTime end = QDateTime::currentDateTime();
DLR_ResultArray *handler = NULL;
DLR_GetAllResults(recognizer, &handler);
std::vector<MRZInfo> all;
QString out = "Elapsed time: " + QString::number(start.msecsTo(end)) + "ms\n\n";
DLR_Result **results = handler->results;
for (int ri = 0; ri < handler->resultsCount; ++ri)
{
    DLR_Result* result = handler->results[ri];
    int lCount = result->lineResultsCount;
    if (lCount < 2)
    {
        continue;
    }

    DLR_LineResult *l1 = result->lineResults[0];
    DLR_LineResult *l2 = result->lineResults[1];

    string line1 = l1->text;
    string line2 = l2->text;
    if (line1.length() != 44 || line2.length() != 44)
    {
        continue;
    }
    if (line1[0] != 'P')
        continue;

    MRZInfo info;

    DM_Point *points = l1->location.points;
    int x1 = points[0].x, y1 = points[0].y;
    int x2 = points[1].x, y2 = points[1].y;
    int x3 = points[2].x, y3 = points[2].y;
    int x4 = points[3].x, y4 = points[3].y;
    DM_Point *points2 = l2->location.points;
    int xx1 = points2[0].x, yy1 = points2[0].y;
    int xx2 = points2[1].x, yy2 = points2[1].y;
    int xx3 = points2[2].x, yy3 = points2[2].y;
    int xx4 = points2[3].x, yy4 = points2[3].y;

    // Type
    string tmp = "Type: ";
    tmp.insert(tmp.length(), 1, line1[0]);
    out += QString::fromStdString(tmp) + "\n";

    // Issuing country
    tmp = "Issuing country: "; line1.substr(2, 5);
    tmp += line1.substr(2, 3);      
    out += QString::fromStdString(tmp) + "\n";

    // Surname
    int index = 5;
    tmp = "Surname: ";
    for (; index < 44; index++)
    {
        if (line1[index] != '<')
        {
            tmp.insert(tmp.length(), 1, line1[index]);
        }
        else 
        {
            break;
        }
    }
    out += QString::fromStdString(tmp) + "\n";

    // Given names
    tmp = "Given Names: ";
    index += 2;
    for (; index < 44; index++)
    {
        if (line1[index] != '<')
        {
            tmp.insert(tmp.length(), 1, line1[index]);
        }
        else 
        {
            tmp.insert(tmp.length(), 1, ' ');
        }
    }
    out += QString::fromStdString(tmp) + "\n";

    // Passport number
    tmp = "Passport number: ";
    index = 0;
    for (; index < 9; index++)
    {
        if (line2[index] != '<')
        {
            tmp.insert(tmp.length(), 1, line2[index]);
        }
        else 
        {
            break;
        }
    }
    out += QString::fromStdString(tmp) + "\n";

    // Nationality
    tmp = "Nationality: ";
    tmp += line2.substr(10, 3);
    out += QString::fromStdString(tmp) + "\n";

    // Date of birth
    tmp = line2.substr(13, 6);
    tmp.insert(2, "/");
    tmp.insert(5, "/");
    tmp = "Date of birth (YYMMDD): " + tmp;
    out += QString::fromStdString(tmp) + "\n";

    // Sex
    tmp = "Sex: ";
    tmp.insert(tmp.length(), 1, line2[20]);
    out += QString::fromStdString(tmp) + "\n";

    // Expiration date of passport
    tmp = line2.substr(21, 6);
    tmp.insert(2, "/");
    tmp.insert(5, "/");
    tmp = "Expiration date of passport (YYMMDD): " + tmp;
    out += QString::fromStdString(tmp) + "\n";

    // Personal number
    if (line2[28] != '<')
    {
        tmp = "Personal number: ";
        for (index = 28; index < 42; index++)
        {
            if (line2[index] != '<')
            {
                tmp.insert(tmp.length(), 1, line2[index]);
            }
            else 
            {
                break;
            }
        }
        out += QString::fromStdString(tmp) + "\n";
    }

    info.text = out;
    info.x1 = x1;
    info.y1 = y1;
    info.x2 = x2;
    info.y2 = y2;
    info.x3 = x3;
    info.y3 = y3;
    info.x4 = x4;
    info.y4 = y4;
    info.xx1 = xx1;
    info.yy1 = yy1;
    info.xx2 = xx2;
    info.yy2 = yy2;
    info.xx3 = xx3;
    info.yy3 = yy3;
    info.xx4 = xx4;
    info.yy4 = yy4;

    all.push_back(info);
}

DLR_FreeResults(&handler);
surface->appendResult(all);
Enter fullscreen mode Exit fullscreen mode

How to Build the Qt CMake Project

The CMake build commands are a bit different between Windows and Linux:

# Windows
mkdir build
cd build
cmake -G "MinGW Makefiles" ..
cmake --build .
MRZRecognizer.exe

# Linux
mkdir build
cd build
cmake ..
cmake --build .
./MRZRecognizer
Enter fullscreen mode Exit fullscreen mode

Running Passport Scanner

When running the program, you need to enter a valid license key:

mrz SDK license

Then you can scan passport information from static images or webcam.

passport scanner for Windows and Linux

Source Code

https://github.com/yushulx/cmake-cpp-barcode-qrcode/tree/main/examples/qt_mrz

Image
15 Top Qualified Gas Engineer Bloggers You Need To Follow
Image
20 Quotes Of Wisdom About Replacing Lost Car Key
Image
Five Things You Don't Know About Wall Hung Electric Fires
Image
Why We Are In Love With Asbestos Settlement (And You Should, Too!)
Image
Unlocking Security: Your Ultimate Guide to Locksmith Services in Wallsend
Image
10 Websites To Help You To Become An Expert In Key Repair Near Me
Image
An Intermediate Guide For Replacing Lost Keys To A Car
Image
20 Trailblazers Setting The Standard In Non Prescription ADHD Medication UK
Image
optifast. A Pc Optimization Software for everyone !

gaming, optimization, fpsgain, lagging
Image
Sécurisation des API avec des Tokens
Image
Could Average Payout For Asbestos Claims Be The Key To 2023's Resolving?
Image
Macam fungsi Dari Judi Online yang Harus Anda Tahu.
Image
good work starts with trying
Image
Situs Judi Beradu Ayam yang Banyak Disenangi Player
Image
The Top Reasons Why People Succeed In The Cost Of Spare Car Key Industry
Image
Forget about axios, this new tool makes automatic data fetching a breeze!

webdev, javascript, alova
Image
Trik Jadi Bandar Judi Online Paling Simpel
Image
Could Corner Sofas Fabric Be The Key To Dealing With 2024?
Image
How To Make A Profitable Traffic Accident Lawyer Near Me Even If You're Not Business-Savvy
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player