OpenSearch

Outputs data to OpenSearch. Compatible with OpenSearch's data ingestion and querying capabilities.

Requirements

To use the OpenSearch output connector, you need to ensure the following prerequisites are met:

1. OpenSearch Cluster: You must have access to an OpenSearch cluster. Ensure that you have the necessary endpoint and credentials to connect to the cluster.

2. Network Access: Ensure that your OpenSearch cluster is network accessible from where the Monad service is running. This may involve configuring network settings, security groups, or firewall rules to allow communication between the Monad service and the OpenSearch cluster.

3. Index Permissions: Ensure that the credentials used to connect to the OpenSearch cluster have the necessary permissions to create, read, and write indices. This includes permissions for bulk indexing operations.

4. Index Configuration: The connector will attempt to create the specified index if it does not already exist. Ensure that your OpenSearch cluster has the necessary settings to allow index creation and configuration. Below is an example of the settings used to create an index.

Example index settings:

{
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    }
}

Functionality

The OpenSearch output connector continuously sends processed data to your specified OpenSearch index. The functionality includes the following key features:

1. Batch Processing: Data is processed and sent to OpenSearch in batches to ensure efficient handling of large volumes of data. The batch size can be configured according to your requirements.

2. Bulk Indexing: Records are indexed in bulk to optimize performance and reduce the overhead of individual indexing operations. This is particularly useful for high-throughput scenarios.

3. Dynamic Index Creation: If the specified index does not exist, the connector will automatically create it using predefined settings. This ensures that your data is always indexed correctly without manual intervention.

4. Error Handling: The connector includes mechanisms to handle errors gracefully, such as logging issues and retrying failed operations. This helps maintain data integrity and reliability.

5. Monitoring and Metrics: The connector provides metrics to monitor the data flow and performance. Metrics such as records in flight, batch processing times, and indexing success rates are available for monitoring.

6. Configuration Flexibility: The connector offers flexible configuration options to customize the indexing process. You can define the index name, batch size, and other parameters to suit your specific use case.

Example data format:

If your data is in JSON format, it will be stored in the specified index with each record represented as a separate document. Below is an example of how a record might be structured:

{
  "id": 1,
  "message": "This is a sample log message",
  "timestamp": "2024-07-02T12:34:56Z"
}

Configuration

The following configuration defines the input parameters. Each field's specifications, such as type, requirements, and descriptions, are detailed below.

Settings

Setting	Type	Required	Description
Username	string	Yes	The username for authenticating with OpenSearch.
URL	string	Yes	The URL of the OpenSearch instance (must start with https).
Insecure Skip Verify	boolean	No	Whether to skip TLS certificate verification (not recommended for production).
Index	string	Yes	The name of the OpenSearch index to use.

Secrets

Secret	Type	Required	Description
Password	string	Yes	The password for authenticating with OpenSearch.