What is Data Prepper in AWS Ingestion Pipeline?

 Data Prepper is an open-source, server-side data collector and streaming ETL engine that powers the OpenSearch Ingestion pipeline. It accepts, filters, transforms, enriches, and routes large-scale observability data (logs, traces, and metrics) before it gets indexed into your cluster. [1, 2]

Core Components

Data Prepper pipelines are configured as customizable streaming graphs. Every pipeline is built with three main elements: [1, 2, 3]

• Sources: The input interfaces that receive or pull data. Common sources include OpenTelemetry (OTel) collectors, Apache Kafka, or Amazon S3. [1]
• Processors: The data manipulation engines. They are used to filter out noise, mask sensitive data for compliance, parse formats (like using the grok parser), or enrich events with geolocation and metadata. [1, 2, 3, 4, 5]
• Sinks: The destinations for your data. Typically, the sink will be an Amazon OpenSearch Service domain or an Amazon OpenSearch Serverless collection, though it can also route to standard output for debugging. [1, 2]

Primary Use Cases

• Trace Analytics: It handles high-volume distributed tracing data, allowing developers to visualize event flows and pinpoint performance issues. [1, 2]
• Log Ingestion: It normalizes messy, unstructured application and system logs into structured, actionable insights for querying in OpenSearch Dashboards. [1, 2]
• Cost Optimization & Security: It applies smart sampling, deduplication, and PII redaction to reduce overall storage costs and ensure data compliance before indexing. [1]