How We Built On-Device Screenshot OCR with Zero Cloud Latency

When you capture a screenshot of an error traceback, a presentation slide, or a design mockup, your natural instinct is often to extract text from it.

Traditional tools either force you to retype text manually or upload your images to third-party cloud Vision APIs — introducing network latency, API costs, and serious privacy risks when screenshots contain private code or credentials.

In Qlip, we built On-Device Screenshot OCR. Whenever you copy an image or capture a screenshot on macOS, Windows, iOS, or Android, Qlip automatically extracts all text in the background in under 40 milliseconds — entirely locally.

Here is how we engineered it.

---

1. Native OS Acceleration over Generic Models

Instead of shipping a heavy multi-hundred megabyte Tesseract OCR binary or ONNX model inside our app bundle, Qlip leverages native platform neural engines:

macOS & iOS: Vision Framework (VNRecognizeTextRequest)

On Apple platforms, Qlip invokes Apple’s native Vision framework directly via Swift bindings. By configuring VNRequestTextRecognitionLevel.fast with automatic language detection, text is extracted in 12 to 25 milliseconds utilizing Apple Silicon’s Neural Engine.

// Swift Native Bridge snippet
import Vision

func extractTextFromImage(cgImage: CGImage) async throws -> String {
    return try await withCheckedThrowingContinuation { continuation in
        let request = VNRecognizeTextRequest { request, error in
            guard let observations = request.results as? [VNRecognizedTextObservation] else {
                continuation.resume(returning: "")
                return
            }
            let recognizedStrings = observations.compactMap { $0.topCandidates(1).first?.string }
            continuation.resume(returning: recognizedStrings.joined(separator: "\n"))
        }
        request.recognitionLevel = .fast
        request.usesLanguageCorrection = true
        
        let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
        try? handler.perform([request])
    }
}

Windows: Windows.Media.Ocr Engine

On Windows 10/11, Qlip bridges directly to the WinRT OcrEngine API. Windows leverages hardware-accelerated WinML models built into the OS, extracting text cleanly without requiring external DLL runtime dependencies.

Android: Google Mobile Vision ML Kit (Local Runtime)

On Android, Qlip utilizes Google ML Kit’s bundlable text recognition runtime, executing text recognition on CPU/NPU local delegates.

---

2. Background Worker Pipeline

To ensure clipboard operations remain zero-lag, OCR processing is executed asynchronously off the main UI thread:

[User Copies Image / Screenshot]
               |
               v
    +----------------------+
    | Main Clipboard Loop  | ---> [Store Raw Image in Local SQLite Cache]
    +----------+-----------+
               |
               | (Async Worker Channel)
               v
    +----------------------+
    | Native OCR Pipeline  | ---> [Extract Recognized Strings]
    +----------+-----------+
               |
               v
    +----------------------------------------------+
    | Index Strings into Local SQLite FTS5 Engine  |
    +----------------------------------------------+

1. Instant Clipboard Store: As soon as Cmd+C or Ctrl+C occurs, Qlip writes the raw image payload to the local SQLite database and pushes the clip to paired devices immediately.
2. Async Worker Channel: The image buffer is dispatched to a background worker thread.
3. Text Recognition: The OS-native OCR engine processes the image.
4. FTS5 Indexing: Extracted text strings are attached as full-text search metadata in SQLite (FTS5).

---

3. Instant Search & Recall

Because extracted text is indexed locally into SQLite’s Full-Text Search index, you can search for words inside a screenshot taken weeks ago:

-- SQLite FTS5 Instant Search Query
SELECT clip_id, payload_type, ocr_text, created_at 
FROM clipboard_history 
WHERE ocr_text MATCH 'InternalServerError' 
ORDER BY created_at DESC;

When you type InternalServerError into Qlip’s fuzzy search popup:

• The screenshot appears in your search results instantly.
• One click extracts and copies the clean text string directly to your clipboard.

---

Privacy & Performance Benefits

• Zero Cloud Latency: Text is searchable within 40ms of taking a screenshot.
• 100% Offline Capability: Works perfectly on airplanes or in secure air-gapped network environments.
• Zero Privacy Leakage: Your sensitive screenshots, mockups, and private documents never touch an external cloud server.

On-device OCR is available out of the box in Qlip for macOS, Windows, iOS, and Android.