How can I extract data from files?

[bolodecenouracomcafe]

Hello, community,

I have been using the Ingest Attachment plugin to ingest my PDF (and other) files, and I would like to migrate to Data Prepper (Amazon OpenSearch Ingestion) for the ingestion process.

How can I extract data from files with Data Prepper? Is there a processor to extract data from files?

[nateynateynate]

Unfortunately, at the moment there is no such feature for Amazon OpenSearch Ingestion. I'd suggest both filing a feature request and perhaps checking out places like https://www.aryn.ai/ that offer specialized solutions for this kind of thing. You could potentially use Aryn to generate the json which would be the "Extract and Transform" part of the ETL pipeline and then have Data Prepper grab the final json, but the actual parsing of the unstructured data in your PDF's can't be handled by Data Prepper by itself.

[bolodecenouracomcafe]

Adding a new supplier is not possible at this time. I will open a feature request. I think it's a common use case for clients who want to migrate from their own ingestion application to Data Prepper and need to extract content from the files. The Ingest-attachment plugin has been available for a long time, so I think many people have used it.