Is it possible to Ingest non JSON content to OpenSearch? How Modern document ingestion works in OpenSearch?

 In OpenSearch, the primary unit of storage is a document, and **all documents must be in JSON format** when they are indexed.

However, this does not mean you are limited to only manual JSON input. You can certainly index unstructured data like PDFs, Word documents, or raw logs, but they must undergo a "transformation" or "ingestion" process to become JSON-compatible before they are stored.

---

### 1. The Core Requirement: JSON

OpenSearch is a distributed search and analytics engine that relies on a structured schema (mappings) to make data searchable. Every document ingested into the engine is stored as a JSON object, which allows OpenSearch to parse the fields, index them, and make them queryable.

### 2. How to Handle Unstructured Data (PDFs, Logs, etc.)

Because you cannot simply "upload" a PDF file directly into an OpenSearch index, you must use an intermediary process to extract the content and convert it into a JSON document.

Here is how you typically handle unstructured data:

* **Extraction & Parsing:** You use external tools or libraries to extract the raw text (or metadata) from your files (e.g., using libraries like Apache Tika, Python's `PyPDF2`, or specialized OCR services).

* **Transformation:** Once the text is extracted, you structure it into a JSON object. For example:

```json

{

  "filename": "annual_report.pdf",

  "content": "Full text extracted from the PDF...",

  "author": "Company Name",

  "timestamp": "2026-07-12T11:12:00Z"

}

```

* **Ingestion Pipeline:** OpenSearch has built-in **Ingest Pipelines** with processors (like `grok`, `csv`, or `json`) that can help clean, parse, and restructure incoming data on the fly.

* **Modern Approaches (Embeddings/AI):** For truly unstructured content (like large documents or images), a common modern pattern is to:

1. Chunk the document text.

2. Use an embedding model to turn that text into "vector embeddings."

3. Store these vectors in OpenSearch using the **k-NN (k-Nearest Neighbors) plugin**, which allows for semantic "meaning-based" search rather than just keyword matching.

### 3. Ways to Ingest Data

Depending on your needs, you can use different methods to get data into OpenSearch:

* **REST API:** Send individual JSON documents using `PUT` or `POST` requests or `_bulk` API calls.

* **Data Prepper:** A server-side data collector (part of the OpenSearch project) that acts as an ETL (Extract, Transform, Load) tool. It can ingest logs, traces, and other data, transform them, and push them to OpenSearch.

* **Logstash/Fluentd:** Common industry tools that can watch files (like server logs), parse them, and forward them as JSON documents to OpenSearch.

* **Language Clients:** If you are writing an application, official OpenSearch clients (for Python, Java, JavaScript, etc.) handle the serialization of your objects into JSON automatically.

---

**Summary**

* **Can I store PDFs/Unstructured data?** Yes, but they must be converted to JSON first.

* **Is it direct?** No, you need a pipeline to extract the text and map it to a JSON schema.

* **How do I get started?** If you are working with files, start by creating a script that parses the files into JSON objects and uses the Bulk API to index them. If you are dealing with logs, look into **Data Prepper**.

Would you like more details on how to set up an Ingest Pipeline to parse specific types of logs or documents?