# Inputs Ingests security data from upstream sources like cloud audit logs, vulnerability scanners, identity providers, and EDR platforms. ## Key Features - Seamless integration with multiple data sources - Automated authentication and data retrieval - Initial formatting for smooth pipeline integration Each input connector is designed to interface with a specific data source. These connectors handle authentication, data retrieval, and initial formatting to ensure smooth integration with our data processing pipeline. ## What You'll Find In the following pages, you'll find: - Configuration instructions for each input connector - Supported data types and formats - Authentication methods and requirements ## Concept: Backfilling data from Inputs The Backfill Start Time setting allows you to control the historical starting point for data collection. This determines how far back in time the system will fetch data during the initial sync. ### How It Works #### Initial Sync Behavior When you first configure a data source: - **With Backfill Start Time**: The system fetches all data from the specified date forward to the present - **Without Backfill Start Time**: No historical data is collected. The system only ingests records produced from the moment of the pipeline's first sync onward ### Subsequent Syncs After the initial sync completes: - All future syncs are **incremental**, collecting only new data since the last successful sync - The Backfill Start Time setting is no longer used - The system automatically tracks its position and continues from where it left off ### Configuration ***Note***: Backfill start time may not exist in every single input connector. Please review the docs of the specific input to verify. #### Format The Backfill Start Time must be provided in **ISO 8601 format**: ``` YYYY-MM-DDTHH:MM:SSZ ``` **Examples:** - `2024-01-01T00:00:00Z` - Start of January 1st, 2024 (UTC) - `2024-06-15T14:30:00Z` - June 15th, 2024 at 2:30 PM (UTC) #### When to Use **Use Backfill Start Time when:** - You only need recent data and want to avoid a lengthy initial sync - You're testing the integration and want to limit data volume - Compliance requirements only mandate retention from a specific date **Skip Backfill Start Time when:** - You only care about data going forward from when the pipeline starts running - You don't need any historical records from before the pipeline was created ### Examples #### Scenario 1: Recent Data Only You only need the last 90 days of audit logs: - Set Backfill Start Time to 90 days ago - Initial sync fetches 90 days of history - Future syncs continue incrementally from the present #### Scenario 2: Compliance Requirement Your organization requires 1 year of audit data: - Set Backfill Start Time to exactly 1 year ago - Initial sync fetches 1 year of history - Future syncs maintain rolling coverage going forward #### Scenario 3: Full Historical Sync You need all available data: - Set Backfill Start Time to a date far enough back to cover the full retention window of the source - Initial sync fetches all available historical data from that date forward - This may take longer depending on data volume #### Scenario 4: Forward-Only Ingestion You only need data going forward from when the pipeline starts: - Leave Backfill Start Time empty - No historical data is fetched - The pipeline only ingests records produced from the moment of its first sync onward ### Important Notes - **One-Time Setting**: Backfill Start Time only affects the first sync; changing it later has no effect - **Time Zone**: Always use UTC timezone in the ISO 8601 format - **Validation**: The system validates the format before starting the sync - **No Data Loss**: Incremental syncs ensure continuous coverage after the initial backfill completes ## Concept: Data Duplication and Loss Our data ingestion system is designed to guarantee **no data loss** while minimizing duplication. State is saved after processing each page of results, enabling safe recovery from any failure scenario. Please read the specific connector's docs for any caveats related to that input data source. ### Guarantees - **Zero Data Loss**: Records are never skipped, even during system failures or interruptions - **Minimal Duplication**: In worst-case scenarios, only the current page being processed may be duplicated - **Automatic Recovery**: The system automatically resumes from the last successful checkpoint ### How It Works ### Checkpoint-Based Processing Data is processed in pages with automatic checkpointing: 1. A page of records is fetched from the source API 2. Each record in the page is processed and sent downstream 3. After the entire page completes successfully, a checkpoint is saved 4. Processing continues with the next page If anything goes wrong, the system restarts from the last saved checkpoint. ### Failure Scenarios ### Normal Operation When everything works correctly: - Each page is processed once - State is saved after each page - No duplication occurs ### Recovery from Failure When a failure occurs during processing: - The system resumes from the last saved checkpoint - Records from the incomplete page are reprocessed - Previously completed pages are never reprocessed **Result**: At most one page of data may be duplicated ### Why Data Is Never Lost Because state is saved after each page completes: - Completed pages are always checkpointed before moving forward - Failed pages are automatically retried on restart - The system never advances past data it hasn't successfully processed **Note**: Please note this documented behavior may differ for specific inputs, and please ensure you review the input's specific documentation to correctly understand intended behavior. ### Design Principles The system is built on these core principles: 1. **Conservative Processing**: When in doubt, the system prefers to duplicate rather than risk losing data 2. **Page-Level Granularity**: Checkpoints happen per-page to balance performance with recovery precision 3. **Idempotent Design**: Downstream systems should be prepared to handle duplicate records gracefully 4. **Automatic Recovery**: No manual intervention is required after failures ### Expected Behavior Under normal conditions, you should see: - **Zero duplicates** during steady-state operation - **Minimal duplicates** (one page maximum) after system restarts or failures - **Complete data coverage** with no gaps in the timeline This makes the system suitable for compliance, audit logging, and other use cases where data completeness is critical.