TrOCR
TrOCR by qualcomm, a image-to-text model with OCR capabilities. Understand and compare OCR features, benchmarks, and capabilities.
Comparison
| Feature | TrOCR | Interfaze |
|---|---|---|
| Input Modalities | image | image, text, audio, video, document |
| Native OCR | Yes | Yes |
| Long Document Processing | No | Yes |
| Language Support | unknown | 162+ |
| Native Speech-to-Text | No | Yes |
| Native Object Detection | No | Yes |
| Guardrail Controls | No | Yes |
| Context Input Size | unknown | 1M |
| Tool Calling | No | Tool calling supported + built in browser, code execution and web search |
OCR Capabilities
| Feature | TrOCR | Interfaze |
|---|---|---|
| Text Bounding Boxes | No | Yes |
| Confidence Scores | No | Yes |
| Dense Image Processing | No | Yes |
| Low Quality Images | No | Yes |
| Handwritten Text | Yes | Yes |
| Charts, Tables & Equations | No | Yes |
Scaling
| Feature | TrOCR | Interfaze |
|---|---|---|
| Scaling | Self-hosted/Provider-hosted with quantization | Unlimited |
View model card on Hugging Face
End-to-end text recognition approach with pre-trained image transformer and text transformer models for both image understanding and wordpiece-level text generation.
This is based on the implementation of TrOCR found here. This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the Qualcomm® AI Hub Models library to export with custom configurations. More details on model performance across various devices, can be found here.
Qualcomm AI Hub Models uses Qualcomm AI Hub Workbench to compile, profile, and evaluate this model. Sign up to run these models on a hosted Qualcomm® device.
Getting Started
There are two ways to deploy this model on your device:
Option 1: Download Pre-Exported Models
Below are pre-exported model assets ready for deployment.
| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.42, ONNX Runtime 1.24.3 | Download |
| QNN_DLC | float | Universal | QAIRT 2.43 | Download |
| TFLITE | float | Universal | QAIRT 2.43, TFLite 2.19.1 | Download |
For more device-specific assets and performance metrics, visit TrOCR on Qualcomm® AI Hub.
Option 2: Export with Custom Configurations
Use the Qualcomm® AI Hub Models Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations
This option is ideal if you need to customize the model beyond the default configuration provided here.
See our repository for TrOCR on GitHub for usage instructions.
Model Details
Model Type: Model_use_case.image_to_text
Model Stats:
- Model checkpoint: trocr-small-stage1
- Input resolution: 320x320
- Number of parameters (decoder): 38.3M
- Model size (decoder) (float): 146 MB
- Number of parameters (encoder): 23.0M
- Model size (encoder) (float): 87.8 MB
Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit |
|---|---|---|---|---|---|---|
| decoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 1.142 ms | 0 - 224 MB | NPU |
| decoder | ONNX | float | Snapdragon® X2 Elite | 1.153 ms | 68 - 68 MB | NPU |
| decoder | ONNX | float | Snapdragon® X Elite | 2.348 ms | 67 - 67 MB | NPU |
| decoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 1.456 ms | 0 - 258 MB | NPU |
| decoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 2.138 ms | 1 - 4 MB | NPU |
| decoder | ONNX | float | Qualcomm® QCS9075 | 2.714 ms | 7 - 16 MB | NPU |
| decoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 1.248 ms | 0 - 252 MB | NPU |
| decoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 1.117 ms | 1 - 212 MB | NPU |
| decoder | QNN_DLC | float | Snapdragon® X2 Elite | 1.6 ms | 7 - 7 MB | NPU |
| decoder | QNN_DLC | float | Snapdragon® X Elite | 2.141 ms | 7 - 7 MB | NPU |
| decoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 1.403 ms | 0 - 232 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 4.216 ms | 7 - 143 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 1.952 ms | 1 - 3 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® SA8775P | 2.833 ms | 7 - 144 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® QCS9075 | 2.549 ms | 7 - 16 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 2.869 ms | 1 - 215 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® SA7255P | 4.216 ms | 7 - 143 MB | NPU |
| decoder | QNN_DLC | float | Qualcomm® SA8295P | 2.934 ms | 7 - 130 MB | NPU |
| decoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 1.236 ms | 0 - 226 MB | NPU |
| decoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 1.097 ms | 0 - 224 MB | NPU |
| decoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 1.36 ms | 0 - 247 MB | NPU |
| decoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 4.235 ms | 0 - 155 MB | NPU |
| decoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 1.939 ms | 0 - 3 MB | NPU |
| decoder | TFLITE | float | Qualcomm® SA8775P | 2.799 ms | 0 - 151 MB | NPU |
| decoder | TFLITE | float | Qualcomm® QCS9075 | 2.514 ms | 0 - 83 MB | NPU |
| decoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 2.575 ms | 0 - 225 MB | NPU |
| decoder | TFLITE | float | Qualcomm® SA7255P | 4.235 ms | 0 - 155 MB | NPU |
| decoder | TFLITE | float | Qualcomm® SA8295P | 3.039 ms | 0 - 134 MB | NPU |
| decoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 1.164 ms | 0 - 231 MB | NPU |
| encoder | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 4.389 ms | 1 - 178 MB | NPU |
| encoder | ONNX | float | Snapdragon® X2 Elite | 4.863 ms | 48 - 48 MB | NPU |
| encoder | ONNX | float | Snapdragon® X Elite | 12.577 ms | 47 - 47 MB | NPU |
| encoder | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 8.29 ms | 16 - 269 MB | NPU |
| encoder | ONNX | float | Qualcomm® QCS8550 (Proxy) | 11.99 ms | 0 - 60 MB | NPU |
| encoder | ONNX | float | Qualcomm® QCS9075 | 15.758 ms | 15 - 19 MB | NPU |
| encoder | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 5.752 ms | 16 - 190 MB | NPU |
| encoder | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 4.662 ms | 2 - 156 MB | NPU |
| encoder | QNN_DLC | float | Snapdragon® X2 Elite | 5.659 ms | 2 - 2 MB | NPU |
| encoder | QNN_DLC | float | Snapdragon® X Elite | 12.47 ms | 2 - 2 MB | NPU |
| encoder | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 8.248 ms | 0 - 231 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® QCS8275 (Proxy) | 38.122 ms | 2 - 149 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® QCS8550 (Proxy) | 11.819 ms | 2 - 3 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® SA8775P | 14.225 ms | 2 - 149 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® QCS9075 | 16.741 ms | 2 - 12 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® QCS8450 (Proxy) | 21.927 ms | 0 - 345 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® SA7255P | 38.122 ms | 2 - 149 MB | NPU |
| encoder | QNN_DLC | float | Qualcomm® SA8295P | 20.266 ms | 2 - 273 MB | NPU |
| encoder | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 5.765 ms | 2 - 155 MB | NPU |
| encoder | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 4.067 ms | 0 - 145 MB | NPU |
| encoder | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 7.719 ms | 6 - 227 MB | NPU |
| encoder | TFLITE | float | Qualcomm® QCS8275 (Proxy) | 36.775 ms | 7 - 145 MB | NPU |
| encoder | TFLITE | float | Qualcomm® QCS8550 (Proxy) | 10.703 ms | 7 - 9 MB | NPU |
| encoder | TFLITE | float | Qualcomm® SA8775P | 13.375 ms | 7 - 146 MB | NPU |
| encoder | TFLITE | float | Qualcomm® QCS9075 | 13.963 ms | 6 - 65 MB | NPU |
| encoder | TFLITE | float | Qualcomm® QCS8450 (Proxy) | 19.604 ms | 7 - 340 MB | NPU |
| encoder | TFLITE | float | Qualcomm® SA7255P | 36.775 ms | 7 - 145 MB | NPU |
| encoder | TFLITE | float | Qualcomm® SA8295P | 19.123 ms | 7 - 269 MB | NPU |
| encoder | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 5.219 ms | 5 - 153 MB | NPU |
License
- The license for the original implementation of TrOCR can be found here.
References
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
- Source Model Implementation
Community
- Join our AI Hub Slack community to collaborate, post questions and learn more about on-device AI.
- For questions or feedback please reach out to us.