# Backblaze B2 Cloud Storage

Write data to Backblaze B2 Cloud Storage buckets.

## Requirements

To configure Backblaze B2 as an output destination for Monad, complete the following steps:

### Step 1: Create Application Keys in Backblaze B2

1. **Log in to your Backblaze B2 Console**  
   Go to the [Backblaze B2 Console](https://secure.backblaze.com/b2_buckets.htm)

2. **Navigate to App Keys**  
   In the sidebar, click on "App Keys"

3. **Create a New Application Key**  
   Click "Add a New Application Key"

4. **Configure the Key**:
   - **Name**: Give your key a descriptive name (e.g., "Monad Integration")
   - **Capabilities**: Select the following permissions:
     - `listBuckets` - to list available buckets
     - `listFiles` - to list files within the bucket  
     - `writeFiles` - to write data to the bucket
     - `readFiles` - to read file contents (for verification)
   - **Bucket Access**: Choose either:
     - "All" for access to all buckets, or
     - "Specific bucket" and select your target bucket

5. **Create and Store the Key**  
   Click "Create New Key" and **immediately copy both the `keyID` and `applicationKey`** - you won't be able to see the `applicationKey` again

### Step 2: Create or Configure Your B2 Bucket

If you don't already have a bucket:
1. In the B2 Console, click "Create a Bucket"
2. Choose a unique bucket name
3. Select your preferred region (note this for configuration)
4. Configure bucket settings as needed

## Functionality

The output continuously sends data to your specified B2 path, formatted as `prefix/partition/filename.format.compression`, where:
- The partition structure depends on your chosen partition format (simple date, hive compliant, or flat hive compliant)
- Files are created based on batching configuration (record count, data size, or time elapsed)
- Data is compressed using your selected compression method before storage

### Batching Behavior

Monad batches records before sending to B2 based on three configurable limits:
- **Record Count**: Maximum number of records per file (default: 100,000, range: 500-1,000,000)
- **Data Size**: Maximum uncompressed size per file (default: 10 MB, range: 1-25 MB)
- **Time Interval**: Maximum time before flushing a batch (default: 45 seconds, range: 1-60 seconds)

Whichever limit is reached first triggers the batch to be written to B2. This ensures timely delivery while optimizing file sizes for downstream processing.

### Output Formats

The output format depends on your configuration:
- **JSON Array Format**: Records are stored as a standard JSON array
- **JSON Nested Format**: Records are wrapped under your specified key (e.g., `{"records": [...]}`)
- **JSON Line Format**: Each record is on its own line (JSONL format)
- **Delimited Format**: Records in CSV or other delimited formats
- **Parquet Format**: Columnar storage format for efficient analytics

## Configuration

### Settings

| Setting | Type | Required | Default | Description |
|---------|------|----------|---------|-------------|
| B2 Bucket Name | string | Yes | - | The name of the B2 bucket where data will be stored |
| B2 Region | string | Yes | us-west-001 | The B2 region endpoint (e.g., us-west-001, us-west-002, eu-central-003) |
| B2 Object Prefix | string | No | - | An optional prefix for B2 object keys to organize data within the bucket |
| Format Configuration | object | Yes | - | The format configuration for output data - see Format Options below |
| Compression Method | string | Yes | - | The compression method to be applied to the data before storing (e.g., gzip, snappy, none) |
| Partition Format | string | Yes | simple_date | The format for organizing data into partitions within the B2 bucket |
| Batch Configuration | object | No | See defaults below | Controls when batches are written to B2 |

#### Format Options

The output format determines how your data is structured in the storage files. You must configure exactly one format type you can see documentation on formats here: [Formats](./formats/index.mdx).

#### Partition Format Options

1. **Simple Date Format** (`simple date`):
- Structure: `{prefix}/{YYYY}/{MM}/{DD}/{filename}`
- Example: `my-data/2024/01/15/20240115T123045Z-uuid.json.gz`

2. **Hive Compliant Format** (`hive compliant`):
- Structure: `{prefix}/year={YYYY}/month={MM}/day={DD}/{filename}`
- Example: `my-data/year=2024/month=01/day=15/20240115T123045Z-uuid.parquet`
- Use case: Compatible with Athena, Hive, and other query engines

3. **Flat Hive Compliant Format** (`flat hive compliant`):
- Structure: `{prefix}/dt={YYYY-MM-DD}/{filename}`
- Example: `my-data/dt=2024-01-15/20240115T123045Z-uuid.parquet`
- Use case: Single-key Hive-style partitioning for tools expecting a `dt` partition column

All partition formats use UTC time for consistency across different time zones.

#### Batch Configuration

| Setting | Type | Default | Min | Max | Description |
|---------|------|---------|-----|-----|-------------|
| record_count | integer | 100,000 | 500 | 1,000,000 | Maximum number of records per file |
| data_size | integer | 10,485,760 (10 MB) | 1,048,576 (1 MB) | 26,214,400 (25 MB) | Maximum uncompressed data size per file in bytes |
| publish_rate | integer | 45 | 1 | 60 | Maximum seconds before flushing a batch |

### Secrets

| Secret | Type | Required | Description |
|---------|------|----------|-------------|
| Application Key ID | string | Yes | Backblaze B2 Application Key ID for authentication |
| Application Key | string | Yes | Backblaze B2 Application Key for authentication |

## Best Practices

1. **File Size Optimization**: Balance between file size and query performance. Larger files are generally better for analytics workloads and reduce per-request costs.

2. **Compression Selection**:
   - **gzip**: Best compression ratio, slower write speed, excellent for long-term storage
   - **none**: Fastest writes, largest file sizes, use when compression happens downstream

3. **Partition Strategy**:
   - Use `hive compliant` when querying with Athena, Presto, or similar services
   - Use `flat hive compliant` for tools expecting a single `dt=YYYY-MM-DD` partition key
   - Use `simple date` for simpler directory structures or custom processing pipelines

4. **Format Selection**:
   - **Parquet**: Best for analytics, columnar queries, and data warehousing
   - **JSON**: Best for flexibility and human readability
   - **CSV**: Best for compatibility with traditional tools and spreadsheets

5. **Cost Optimization**:
   - Larger batch sizes reduce the number of write calls
   - Use appropriate compression to minimize storage costs
   - Consider B2's lifecycle rules for automatic data archiving

6. **Security Best Practices**:
   - Use application keys with minimal required permissions
   - Regularly rotate application keys
   - Consider bucket-specific keys for different environments

## Troubleshooting

### Common Issues

1. **Authentication Failed**: 
   - Verify your Application Key ID and Application Key are correct
   - Ensure the key has the required capabilities (`listBuckets`, `listFiles`, `writeFiles`, `readFiles`)
   - Check that the key has access to the specified bucket

2. **Bucket Not Found**: 
   - Verify the bucket name is correct and exists
   - Ensure the application key has access to the bucket
   - Check that the bucket is in the specified region

3. **Connection Timeout**: 
   - Verify the region setting matches your bucket's actual region
   - Check network connectivity to Backblaze B2 endpoints
   - Ensure no firewall rules are blocking the connection

4. **Permission Denied**: 
   - Verify the application key has `writeFiles` capability
   - Check that bucket permissions allow the operation
   - Ensure the key is not expired

5. **Parse Errors**: 
   - Ensure the file format setting matches your data structure
   - Verify the record format for JSON files
   - Check that compression setting is supported

6. **Performance Issues**:
   - Consider increasing batch size to reduce API calls
   - Use appropriate compression for your use case
   - Verify region selection for optimal latency

## Related Articles

- [Backblaze B2 Input](/docs/inputs/backblaze/backblaze-b2-cloud-storage.mdx)
- [Formats Documentation](./formats/index.mdx)
- [Backblaze B2 Documentation](https://www.backblaze.com/b2/docs/)