---
title: "Custom Tokenizers for PDF Search on iOS | Nutrient SDK"
canonical_url: "https://www.nutrient.io/guides/ios/memory-and-storage/using-custom-tokenizers/"
md_url: "https://www.nutrient.io/guides/ios/memory-and-storage/using-custom-tokenizers.md"
last_updated: "2026-05-15T17:18:57.913Z"
description: "Learn to enhance PDF search functionality on iOS by implementing custom tokenizers with PSPDFKit. Improve indexing performance and optimize exact matches for better results."
---

# Custom Tokenizers for PDF Search on iOS

PSPDFKit uses SQLite to build the full-text index used in [`PDFLibrary`](https://www.nutrient.io/api/ios/documentation/pspdfkit/pdflibrary) and [`PDFDocumentPickerController`](https://www.nutrient.io/api/ios/documentation/pspdfkitui/pdfdocumentpickercontroller), and also for various other data-saving operations (like the image cache metadata). PSPDFKit doesn’t ship with its own SQLite version, and instead it uses the one that’s already in iOS. PSPDFKit also supports [custom SQLite builds](https://www.nutrient.io/../../miscellaneous/custom-sqlite-library).

By default, [`PDFLibrary`](https://www.nutrient.io/api/ios/documentation/pspdfkit/pdflibrary) uses its own tokenizer, which works well for many languages, including Chinese, Japanese, and Korean (CJK). It also enables searching for related words, e.g. finding “dependencies” when searching for “depending.” This is implemented by the [`PSPDFLibraryPorterTokenizerName`](https://www.nutrient.io/api/ios/documentation/pspdfkit/pspdflibraryportertokenizername).

When should you ship your own build of SQLite?

- When you want better indexing performance

- When you need features only available in a newer version of SQLite

- When you need better performance for exact word or phrase matches

If you rely a lot on exact word or phrase matches, the default tokenizer set by [`PDFLibrary`](https://www.nutrient.io/api/ios/documentation/pspdfkit/pdflibrary) might not be optimal and you should consider switching to a custom one.

By default, PSPDFKit uses a custom tokenizer for building the full-text search (FTS) index that can deal with CJK characters as well. Alternatively, we ship another custom tokenizer, which is referenced by the [`PDFLibrary.UnicodeTokenizerName`](https://www.nutrient.io/api/ios/documentation/pspdfkit/pdflibrary/unicodetokenizername) identifier. This tokenizer is a wrapper around SQLite’s [`unicode61`](https://www.sqlite.org/fts3.html#tokenizer) tokenizer, but it performs full case folding. This is useful in cases where the document being indexed has text like `Straße` and you’d like it to match when searching for `strasse`.

You can also use the [custom tokenizers](https://www.sqlite.org/fts3.html#tokenizer) shipped with SQLite itself, like the `unicode61` or `icu` tokenizers.

| Tokenizer                                            | Minimum FTS Version | Minimum SQLite Version |
| ---------------------------------------------------- | ------------------- | ---------------------- |
| [PSPDFLibraryPorterTokenizerName](https://www.nutrient.io/api/ios/documentation/pspdfkit/pspdflibraryportertokenizername)  | FTS4                | 3.7.4                  |
| [PDFLibrary.UnicodeTokenizerName](https://www.nutrient.io/api/ios/documentation/pspdfkit/pdflibrary/unicodetokenizername) | FTS5                | 3.9.0                  |
| [`unicode61`](https://www.sqlite.org/fts3.html#tokenizer)                     | FTS4                | 3.7.13                 |

Note that simply linking the correct SQLite version with your application isn’t enough: You must ensure that the linked SQLite is built with the correct flags to enable FTS4 or FTS5. Trying to enable a tokenizer on an unsupported FTS version will result in the initialization of [`PDFLibrary`](https://www.nutrient.io/api/ios/documentation/pspdfkit/pdflibrary) failing:

### SWIFT

```swift

do {
    let library = try PDFLibrary(path: PDFLibrary.defaultLibraryPath(), tokenizer: "unicode61")
    let documentPicker = PDFDocumentPickerController(directory: "/path/to/files", includeSubdirectories: true, library: library)
} catch {
    // Handle error.
}

```

### OBJECTIVE-C

```objc

PSPDFLibrary *library = [PSPDFLibrary libraryWithPath:PSPDFLibrary.defaultLibraryPath tokenizer:@"unicode61" error:NULL];
PSPDFDocumentPickerController *documentPicker = [[PSPDFDocumentPickerController alloc] initWithDirectory:@"/path/to/files" includeSubdirectories:YES library:library];

```

Optionally, you can also ship your own version of SQLite. To do so, do the following. In the `PSPDFKit.dmg` you downloaded, you’ll find a current version of SQLite in the Extras folder that’s already prepared to be linked. Add `SQLite.xcodeproj` to your Xcode project, and then add `libSQLite.a` as a Target Dependency and under Link Binary with Libraries. Make sure you don’t link the `libsqlite3.tbd` library.

You’ll have to delete your app, or at least the library file, so that the index is fully rebuilt after a different tokenizer has been set.
---

## Related pages

- [Adding a Custom SQLite Library for PDF Search on iOS](/guides/ios/miscellaneous/custom-sqlite-library.md)
- [Generate PDF Search Previews on iOS](/guides/ios/search/indexed-full-text-search/customize-results.md)
- [Indexing PDF Documents on iOS](/guides/ios/features/indexed-full-text-search.md)
- [Encrypt the PDF Search Database on iOS](/guides/ios/security/encryption-in-pdflibrary.md)
- [Index and Search PDFs Using iOS Spotlight](/guides/ios/search/indexed-full-text-search/spotlight-indexing.md)
- [PDF Search Matching Options on iOS](/guides/ios/search/indexed-full-text-search/matching-options.md)