bug with PDF Smalot

[eyona]

Hello,

When I index large pdf db, I got this error

An unhandled exception has been thrown:
TypeError: Smalot\PdfParser\PDFObject::getTJUsingFontFallback(): Argument #1 ($font) must be of type Smalot\PdfParser\Font, null given, called in /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/PDFObject.php on line 930 and defined in /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/PDFObject.php:527
Stack trace:
#0 /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/PDFObject.php(930): Smalot\PdfParser\PDFObject->getTJUsingFontFallback()
#1 /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/PDFObject.php(668): Smalot\PdfParser\PDFObject->getTextArray()
#2 /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/PDFObject.php(798): Smalot\PdfParser\PDFObject->getText()
#3 /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/PDFObject.php(668): Smalot\PdfParser\PDFObject->getTextArray()
#4 /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/Page.php(232): Smalot\PdfParser\PDFObject->getText()
#5 /opt/nextcloud/apps/fulltextsearch_sql/vendor/smalot/pdfparser/src/Smalot/PdfParser/Document.php(439): Smalot\PdfParser\Page->getText()
#6 /opt/nextcloud/apps/fulltextsearch_sql/lib/Platform/SQLPlatform.php(238): Smalot\PdfParser\Document->getText()
#7 /opt/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(353): OCA\FullTextSearch_SQL\Platform\SQLPlatform->indexDocument()
#8 /opt/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(303): OCA\FullTextSearch\Service\IndexService->indexDocument()
#9 /opt/nextcloud/apps/fulltextsearch/lib/Service/IndexService.php(174): OCA\FullTextSearch\Service\IndexService->indexDocuments()
#10 /opt/nextcloud/apps/fulltextsearch/lib/Command/Index.php(325): OCA\FullTextSearch\Service\IndexService->indexProviderContentFromUser()
#11 /opt/nextcloud/apps/fulltextsearch/lib/Command/Index.php(202): OCA\FullTextSearch\Command\Index->indexProvider()
#12 /opt/nextcloud/3rdparty/symfony/console/Command/Command.php(326): OCA\FullTextSearch\Command\Index->execute()
#13 /opt/nextcloud/core/Command/Base.php(218): Symfony\Component\Console\Command\Command->run()
#14 /opt/nextcloud/3rdparty/symfony/console/Application.php(1078): OC\Core\Command\Base->run()
#15 /opt/nextcloud/3rdparty/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand()
#16 /opt/nextcloud/3rdparty/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun()
#17 /opt/nextcloud/lib/private/Console/Application.php(187): Symfony\Component\Console\Application->run()
#18 /opt/nextcloud/console.php(90): OC\Console\Application->run()
#19 /opt/nextcloud/occ(33): require_once('...')

missing font

This stop indexing process

[jplitza]

Can you identify which PDF caused the error? This seems to be a bug in the PDF parsing library I use, so you could report it there.

But I'll see if I can make the indexing process more robust against PDF parsing errors .

[jplitza]

Wait, that exception should already be caught and properly reported as indexing error of that single document, not affecting any other document's indexing.

How exactly are you starting the indexing process? How exactly is this error displayed?

[eyona]

Hello, I start indexing by cli sudo -u www-data php occ fulltextsearch:live I have a large base of pdf, some of these are exported by adobe indesign Memory: 56 MB ┌─ Indexing ──── │ Action: waiting │ Provider: Files Account: stuff │ Document: 102476 │ Info: application/pdf │ Title: Mediatheque/.... │ Content size: 18998236 └── ┌─ Results ──── │ Result: 325/325 │ Index: files:102476 │ Status: ok │ Message: true │ │ └── ┌─ Errors ──── │ Error: 16/16 │ Index: files:63866 │ Exception: │ Message: An exception occurred while executing a query: SQLSTATE[54000]: Program limit exceeded: 7 ERROR: string is too long for tsvector (1859180 bytes, max 1048575 bytes) And, as you can see, I got a new error, about vector db SQLSTATE[54000]: Program limit exceeded: 7 ERROR: string is too long for tsvector (1859180 bytes, max 1048575 bytes) The platform should handle this case gracefully by either: truncating the text before passing it to to_tsvector(), splitting large documents into multiple chunks, or skipping the document with a clear warning instead of throwing a fatal SQL exception. Best regards

…

Le 28 oct. 2025 à 20:34, Jan-Philipp Litza ***@***.***> a écrit : jplitza left a comment (jplitza/fulltextsearch_sql#24) <#24 (comment)> Wait, that exception should already be caught and properly reported as indexing error of that single document, not affecting any other document's indexing. How exactly are you starting the indexing process? How exactly is this error displayed? — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZZKZXHU2SDVIBTTKIHUTT3Z7AM7AVCNFSM6AAAAACKOAFFQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINJYGE3TCMJSGI>. You are receiving this because you authored the thread.

[eyona]

For your ref, my env run on ubuntu 24.04 LTS, nginx, php8.3-fpm, postgres 16

…

Le 28 oct. 2025 à 20:34, Jan-Philipp Litza ***@***.***> a écrit : jplitza left a comment (jplitza/fulltextsearch_sql#24) <#24 (comment)> Wait, that exception should already be caught and properly reported as indexing error of that single document, not affecting any other document's indexing. How exactly are you starting the indexing process? How exactly is this error displayed? — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZZKZXHU2SDVIBTTKIHUTT3Z7AM7AVCNFSM6AAAAACKOAFFQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTINJYGE3TCMJSGI>. You are receiving this because you authored the thread.

[jplitza]

Sorry for the long delay.

skipping the document with a clear warning instead of throwing a fatal SQL exception.

This is exactly what happens AFAICT. In that interactive occ fulltextsearch:live invocation, you can flip through the errors that were thrown for different files with the h and j keys. Only when that command exits unexpectedly was an exception not caught.

I opened #31 to handle overlong content more gracefully.