Skip to main content

Elasticsearch

Sends processed data to an Elasticsearch cluster. Supports indexing of security events for search and analysis.

Requirements​

To use the Elasticsearch output connector, Monad allows you to connect via one of 4 methods:

  • URL Connection with Password: Requires Elasticsearch URL, username, password, and index name.
  • URL Connection with API Key: Requires Elasticsearch URL, API key, and index name.
  • Cloud ID Connection with Password: Requires Elastic Cloud ID, username, password, and index name.
  • Cloud ID Connection with API Key: Requires Elastic Cloud ID, API key, and index name.

Obtaining Elasticsearch Credentials​

Login into your Elastic Cloud Instance — https://cloud.elastic.co/home, and below are screens in Elastic Cloud where you can find the appropriate credentials.

  1. Finding Your Elasticsearch URL
  1. Generating an API Key
  1. Locating Your Cloud ID
  1. Setting Up Username and Password
  2. Creating an Index

Example index settings:

{
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    }
}

Details​

The Elasticsearch output connector continuously sends processed data to your specified Elasticsearch index. The functionality includes the following key features:

  1. Batch Processing: Data is processed and sent to Elasticsearch in batches to ensure efficient handling of large volumes of data. The batch size can be configured according to your requirements.

  2. Bulk Indexing: Records are indexed in bulk using Elasticsearch's bulk indexing API. This optimizes performance and reduces the overhead of individual indexing operations, especially in high-throughput scenarios.

  3. Dynamic Index Creation: If the specified index does not exist, the connector will automatically create it using predefined settings. This ensures that your data is always indexed correctly without manual intervention.

  4. Error Handling: The connector includes mechanisms to handle errors gracefully, such as logging issues and retrying failed operations. This helps maintain data integrity and reliability.

  5. Monitoring and Metrics: The connector provides metrics to monitor the data flow and performance. Metrics such as records in flight, batch processing times, and indexing success rates are available for monitoring.

  6. Configuration Flexibility: The connector offers flexible configuration options to customize the indexing process. You can define the index name, batch size, number of workers, flush bytes, and flush interval to suit your specific use case.

  7. Cluster Health Monitoring: During initialization, the connector checks the health of the Elasticsearch cluster to ensure it is ready to receive data. This helps in identifying and resolving cluster-related issues early.

Example data format:

If your data is in JSON format, it will be stored in the specified index with each record represented as a separate document. Below is an example of how a record might be structured:

{
  "id": 1,
  "message": "This is a sample log message",
  "timestamp": "2024-07-02T12:34:56Z"
}

Configuration​

The following configuration defines the input parameters. Each field's specifications, such as type, requirements, and descriptions, are detailed below.

Settings​

SettingTypeRequiredDescription
UsernamestringYesThe username for authenticating with Elasticsearch.
Elasticsearch URLstringYesThe URL of the Elasticsearch cluster. Required when connection type is set to 'url'.
Insecure Skip VerifybooleanNoIf set to true, it skips verification of the server's TLS certificate. This is insecure and should only be used for testing purposes.
Index NamestringYesThe name of the Elasticsearch index to write data to. If the index doesn't exist, it will be created automatically.
Authentication TypestringYesThe method of authentication to use with the Elasticsearch cluster. Choose between 'api_key' or 'password'.
Connection TypestringYesThe type of connection to use with Elasticsearch. Choose between 'cloud_id' for Elastic Cloud or 'url' for direct connection.
Cloud IDstringYesThe Cloud ID for connecting to an Elastic Cloud deployment. Required when connection_type is set to 'cloud_id'.

Secrets​

SecretTypeRequiredDescription
PasswordstringYesThe password for authenticating with ElasticSearch.
API KeystringYesAPI key for authenticating with the Elasticsearch cluster. Required when auth type is set to 'api_key'.