Because OpenSearch is a search and analytics engine, it expects data to be sent via APIs or ingestion pipelines. Here are the standard ways to upload data:
Bulk API (REST API):
For small to medium-sized manual uploads (like a JSON file), you typically use the _bulk API. You can run these commands from your local machine using tools like curl or Postman.
Process: You format your data into a specific format called NDJSON (Newline Delimited JSON) and send it to your OpenSearch domain endpoint.
OpenSearch Ingestion Pipelines (Managed):
For production scenarios, AWS provides Amazon OpenSearch Ingestion. This is a managed, serverless service that can automatically pull data from sources like Amazon S3, Kinesis, DynamoDB, or Kafka and index it into your OpenSearch domain.
Client Libraries (SDKs):
If you are building an application, you should use the official OpenSearch client libraries (available for languages like Python, Java, JavaScript, etc.) to programmatically index documents as your application generates them.
Data Prepper:
This is an open-source data collector that you can run on your own infrastructure to filter, enrich, and transform data before it reaches OpenSearch.
Can you use OpenSearch Dashboards for this?
No, you generally cannot use the OpenSearch Dashboards UI to "upload" a file to index it.
Dashboards for Management: You can use Dashboards to create index patterns, manage existing indexes, and run queries to verify that data has been uploaded correctly.
Dev Tools: Inside Dashboards, there is a "Dev Tools" console where you can manually type out and execute API requests, including bulk index commands. While this technically allows you to input data through the UI, it is not an "upload" feature and is meant for testing and development, not for batch processing files.
No comments:
Post a Comment