# JSON Schema Format for Parquet Output

This document explains how to define a Parquet schema using a JSON format for any Monad output component that supports parquet format.

## Overview

The JSON schema format allows you to define complex Parquet schemas including nested structures, lists, maps, and various data types with their respective encodings and compression options.

## Basic Structure

A Parquet schema in JSON format consists of a root element and a collection of fields:

```json
{
  "Tag": "name=root_name, repetitiontype=REQUIRED",
  "Fields": [
    // field definitions go here
  ]
}
```

## Field Definition

Each field is defined as a JSON object with at least a `Tag` property. For nested structures, a field may also contain a `Fields` array:

```json
{
  "Tag": "name=field_name, type=DATA_TYPE, repetitiontype=REPETITION_TYPE, [additional attributes]"
}
```

### Common Tag Attributes

| Attribute | Description | Example |
|-----------|-------------|---------|
| `name` | Field name in the Parquet file (required) | `name=customer_id` |
| `inname` | Input field name in your source data (optional) | `inname=CustomerID` |
| `type` | Parquet data type (required) | `type=INT64` |
| `repetitiontype` | Field repetition type (required) | `repetitiontype=REQUIRED` |
| `convertedtype` | Logical type for the field (optional) | `convertedtype=UTF8` |
| `encoding` | Encoding method (optional) | `encoding=PLAIN_DICTIONARY` |
| `omitstats` | Skip stats for this field (optional) | `omitstats=true` |

## Primitive Data Types

| Type | Description | Go Type Mapping |
|------|-------------|----------------|
| `BOOLEAN` | Boolean value | `bool` |
| `INT32` | 32-bit integer | `int32` |
| `INT64` | 64-bit integer | `int64` |
| `INT96` | 96-bit integer (deprecated) | `string` |
| `FLOAT` | Single precision (32-bit) floating point | `float32` |
| `DOUBLE` | Double precision (64-bit) floating point | `float64` |
| `BYTE_ARRAY` | Array of bytes | `string` |
| `FIXED_LEN_BYTE_ARRAY` | Fixed-length byte array | `string` (requires `length` attribute) |

## Logical Types

Logical types provide additional semantic information for primitive types:

| Logical Type | Primitive Type | Attributes | Example |
|--------------|----------------|------------|---------|
| `UTF8` | `BYTE_ARRAY` | `convertedtype=UTF8` | `"Tag": "name=username, type=BYTE_ARRAY, convertedtype=UTF8"` |
| `INT_8/16/32/64` | `INT32/INT64` | `convertedtype=INT_8` | `"Tag": "name=small_int, type=INT32, convertedtype=INT_8"` |
| `UINT_8/16/32/64` | `INT32/INT64` | `convertedtype=UINT_16` | `"Tag": "name=unsigned, type=INT32, convertedtype=UINT_16"` |
| `DATE` | `INT32` | `convertedtype=DATE` | `"Tag": "name=birth_date, type=INT32, convertedtype=DATE"` |
| `TIME_MILLIS` | `INT32` | `convertedtype=TIME_MILLIS` | `"Tag": "name=event_time, type=INT32, convertedtype=TIME_MILLIS"` |
| `TIME_MICROS` | `INT64` | `convertedtype=TIME_MICROS` | `"Tag": "name=precise_time, type=INT64, convertedtype=TIME_MICROS"` |
| `TIMESTAMP_MILLIS` | `INT64` | `convertedtype=TIMESTAMP_MILLIS` | `"Tag": "name=created_at, type=INT64, convertedtype=TIMESTAMP_MILLIS"` |
| `TIMESTAMP_MICROS` | `INT64` | `convertedtype=TIMESTAMP_MICROS` | `"Tag": "name=modified_at, type=INT64, convertedtype=TIMESTAMP_MICROS"` |
| `DECIMAL` | Various | `convertedtype=DECIMAL, scale=N, precision=M` | `"Tag": "name=price, type=INT32, convertedtype=DECIMAL, scale=2, precision=9"` |

## Complex Types

### Lists

```json
{
  "Tag": "name=items, type=LIST, repetitiontype=REQUIRED",
  "Fields": [
    {"Tag": "name=element, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}
  ]
}
```

### Maps

```json
{
  "Tag": "name=properties, type=MAP, repetitiontype=REQUIRED",
  "Fields": [
    {"Tag": "name=key, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
    {"Tag": "name=value, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}
  ]
}
```

### Nested Structures

```json
{
  "Tag": "name=address, repetitiontype=REQUIRED",
  "Fields": [
    {"Tag": "name=street, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
    {"Tag": "name=city, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
    {"Tag": "name=zip, type=INT32, repetitiontype=REQUIRED"}
  ]
}
```

### Repeated Fields

```json
{"Tag": "name=phone_numbers, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REPEATED"}
```

## Repetition Types

| Type | Description |
|------|-------------|
| `REQUIRED` | The field must be present (non-null) |
| `OPTIONAL` | The field can be null |
| `REPEATED` | The field can appear multiple times (similar to an array) |

## Encoding Options

| Type | Support | Description |
|------|---------|-------------|
| `PLAIN` | All types | Default encoding |
| `PLAIN_DICTIONARY` | All types | Dictionary-based encoding for repeated values |
| `DELTA_BINARY_PACKED` | Integer types | Efficient for sequences with small deltas |
| `DELTA_BYTE_ARRAY` | BYTE_ARRAY, UTF8 | Efficient for strings with common prefixes |
| `DELTA_LENGTH_BYTE_ARRAY` | BYTE_ARRAY, UTF8 | Efficient for strings with similar lengths |

## Complete Example

```json
{
  "Tag": "name=customer_record, repetitiontype=REQUIRED",
  "Fields": [
    {"Tag": "name=id, type=INT64, repetitiontype=REQUIRED"},
    {"Tag": "name=name, type=BYTE_ARRAY, convertedtype=UTF8, encoding=PLAIN_DICTIONARY, repetitiontype=REQUIRED"},
    {"Tag": "name=email, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=OPTIONAL"},
    {"Tag": "name=signup_date, type=INT32, convertedtype=DATE, repetitiontype=REQUIRED"},
    {"Tag": "name=active, type=BOOLEAN, repetitiontype=REQUIRED"},
    
    {
      "Tag": "name=address, repetitiontype=OPTIONAL",
      "Fields": [
        {"Tag": "name=street, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
        {"Tag": "name=city, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
        {"Tag": "name=state, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
        {"Tag": "name=zip, type=INT32, repetitiontype=REQUIRED"}
      ]
    },
    
    {
      "Tag": "name=purchases, type=LIST, repetitiontype=OPTIONAL",
      "Fields": [
        {
          "Tag": "name=element, repetitiontype=REQUIRED",
          "Fields": [
            {"Tag": "name=product_id, type=INT64, repetitiontype=REQUIRED"},
            {"Tag": "name=price, type=INT32, convertedtype=DECIMAL, scale=2, precision=9, repetitiontype=REQUIRED"},
            {"Tag": "name=quantity, type=INT32, repetitiontype=REQUIRED"}
          ]
        }
      ]
    },
    
    {
      "Tag": "name=properties, type=MAP, repetitiontype=OPTIONAL",
      "Fields": [
        {"Tag": "name=key, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"},
        {"Tag": "name=value, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}
      ]
    },
    
    {"Tag": "name=tags, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REPEATED"}
  ]
}
```

## Tips and Best Practices

1. Use `inname` when your input field names differ from how you want them stored in Parquet
2. For large fields with many different values, avoid `PLAIN_DICTIONARY` encoding as it can consume excessive memory
3. For large array values where stats aren't useful, use `omitstats=true` to reduce file size
4. Use the appropriate repetition type:
    - `REQUIRED` for non-nullable fields
    - `OPTIONAL` for nullable fields
    - `REPEATED` for array-like fields
5. Field names beginning with uppercase and lowercase letters are treated as different fields
6. Avoid using `PARGO_PREFIX_` as a name prefix as it's reserved