# Formats

# Output Formats

Monad supports multiple output formats to meet diverse data integration needs. This guide helps you choose the right format for your use case and provides links to detailed configuration documentation for each format.

## Supported Formats Overview

Monad currently supports three primary output format families:

| Format | Description | File Extensions | Best For |
|--------|-------------|-----------------|----------|
| **JSON** | Flexible text-based format with three variants | `.json`, `.jsonl` | APIs, web services, streaming, human-readable data |
| **Delimited** | Tabular formats like CSV and TSV | `.csv` | Spreadsheets, traditional analytics tools, data exchange |
| **Parquet** | Columnar binary format optimized for analytics | `.parquet` | Data warehouses, big data analytics, long-term storage |

## Format Comparison

### Performance Characteristics

| Format | Write Speed | Read Speed | Compression | File Size |
|--------|------------|------------|-------------|-----------|
| JSON Array | Medium | Medium | Good with gzip | Medium |
| JSON Line | Fast | Fast | Good with gzip | Medium |
| JSON Nested | Medium | Medium | Good with gzip | Medium |
| Delimited (CSV) | Fast | Fast | Excellent with gzip | Small |
| Parquet | Slow | Very Fast | Built-in (excellent) | Very Small |

### Feature Support

| Feature | JSON | Delimited | Parquet |
|---------|------|-----------|---------|
| Human Readable | ✅ Yes | ✅ Yes | ❌ No (binary) |
| Schema Evolution | ✅ Flexible | ⚠️ Limited | ✅ Yes |
| Nested Data | ✅ Full support | ❌ No | ✅ Full support |
| Data Types | ⚠️ Basic | ⚠️ Strings only | ✅ Rich type system |
| Query Performance | ⚠️ Scan entire file | ⚠️ Scan entire file | ✅ Columnar access |
| Streaming Support | ✅ Yes (line format) | ✅ Yes | ❌ No |

## Choosing the Right Format

### Use JSON When:
- **Integrating with REST APIs** or web services
- **Human readability** is important
- Your data has **nested structures** or varying schemas
- You need **streaming capabilities** (use line format)
- Working with **document-oriented databases**

### Use Delimited (CSV/TSV) When:
- Importing data into **spreadsheet applications**
- Working with **legacy systems** or traditional BI tools
- Your data is **purely tabular** with consistent columns
- You need the **smallest file sizes** when compressed
- **Simplicity** is more important than features

### Use Parquet When:
- Building a **data warehouse** or data lake
- Performing **analytical queries** on large datasets
- You need **optimal query performance**
- **Long-term storage** with efficient compression is required
- Working with **Apache Spark**, **Athena**, or similar big data tools

## Quick Configuration Examples

### JSON Array Format
```json
{
  "format_config": {
    "format": "json",
    "json": {
      "type": "array"
    }
  }
}
```

### CSV Format
```json
{
  "format_config": {
    "format": "delimited",
    "delimited": {
      "delimiter": ",",
      "headers": ["id", "name", "value"]
    }
  }
}
```

### Parquet Format
```json
{
  "format_config": {
    "format": "parquet",
    "parquet": {
      "schema": "{\"Tag\": \"name=data\", \"Fields\": [{\"Tag\": \"name=id, type=INT64, repetitiontype=REQUIRED\"}]}"
    }
  }
}
```

## Format-Specific Documentation

For detailed configuration options and advanced usage:

- **[JSON Format](./json.mdx)** - Array, nested, and line-delimited JSON configurations
- **[Delimited Format](./delimited.mdx)** - CSV, TSV, and custom delimiter configurations
- **[Parquet Format](./parquet.mdx)** - Schema definition and columnar storage optimization

## Best Practices

1. **Consider Your Downstream Systems**: Choose formats that your consuming applications can efficiently process.

2. **Balance Readability and Performance**: JSON offers readability, Parquet offers performance, and CSV offers compatibility.

3. **Think About Schema Evolution**: If your data structure changes frequently, JSON provides the most flexibility.

4. **Compression Matters**: All formats support compression, but Parquet has it built-in and typically achieves the best ratios.

5. **Test with Real Data**: Performance characteristics can vary based on your specific data patterns and volumes.

## Common Patterns

### ETL Pipelines
- **Extract**: Use JSON or CSV for initial data extraction
- **Transform**: Process in memory or with streaming tools
- **Load**: Use Parquet for final storage in data warehouses

### Real-time Streaming
- Use **JSON Line format** for append-friendly streaming
- Enable minimal batching for near real-time delivery
- Consider compression trade-offs for latency

### Analytics Workloads
- Use **Parquet** with Hive-compatible partitioning
- Define schemas that optimize for your query patterns
- Leverage columnar benefits for aggregation queries

## Need Help?

If you're unsure which format to choose, consider these questions:

1. What system will consume this data?
2. How important is human readability?
3. What's the expected data volume?
4. Do you need to support complex nested structures?
5. Will the schema change frequently?

The answers to these questions will guide you toward the optimal format for your use case.