# Using One Input in Multiple Pipelines

You can attach the same input to more than one pipeline, either within one organization or by [sharing it with your sub-orgs](./resource-sharing.mdx).
Whether every pipeline then receives all of the data depends on how that input collects it.
For most inputs each pipeline gets a full, independent copy, but inputs that read from a queue split the data between pipelines instead.

## How it works

An input is a reusable configuration, not a shared connection.
Each pipeline that uses it runs its own copy of the connector, with its own checkpoint, and nothing is coordinated between pipelines.
What that means in practice depends on the kind of input:

| Input kind | Examples | What each pipeline receives |
| ---------- | -------- | --------------------------- |
| Pulls from an API | Most SaaS connectors | A full, independent copy. Each pipeline makes its own requests to the source. |
| Polls cloud storage | [S3](../inputs/aws/s3.mdx), [Azure Blob Storage](../inputs/microsoft/microsoft-azure-blob-storage.mdx), [Google Cloud Storage](../inputs/google/google-cloud-storage.mdx) | A full copy, once both pipelines' checkpoints cover the same time range. See [Polling cloud storage](#polling-cloud-storage). |
| Reads from a queue or subscription | [SQS](../inputs/aws/aws-sqs.mdx), [SQS S3](../inputs/aws/aws-sqs-s3.mdx), [SQS S3 CloudTrail](../inputs/aws/aws-sqs-s3-cloudtrail.mdx), [SQS S3 GuardDuty](../inputs/aws/aws-sqs-s3-guardduty.mdx), [Security Hub Findings](../inputs/aws/aws-security-hub-findings.mdx), [Falcon Data Replicator](../inputs/crowdstrike/crowdstrike-falcon-data-replicator.mdx), [Azure Queue Storage](../inputs/microsoft/azure-queue-storage.mdx), [Pub/Sub](../inputs/google/google-pubsub.mdx), [Kafka Messages](../inputs/kafka/kafka-messages.mdx), [Confluent Messages](../inputs/confluent/confluent-messages.mdx) | **A share of the data, not a copy.** Each message goes to exactly one pipeline. See [Inputs that read from a queue](#inputs-that-read-from-a-queue). |
| Reads from Azure Event Hubs | [Azure Event Hubs](../inputs/microsoft/azure-event-hubs.mdx) | A full copy, within limits. See [Azure Event Hubs](#azure-event-hubs). |
| Receives HTTP requests | [HTTP](../inputs/monad/monad-http.mdx) | Only what is sent to that pipeline's own ingest endpoint. |
| Receives syslog | [Syslog](../inputs/monad/monad-syslog.mdx) | Depends on how the sender identifies the pipeline. See [Syslog](#syslog). |

## Inputs that read from a queue

:::warning[Pipelines sharing a queue split the data]
Two pipelines that read from the same queue compete for its messages.
Each message is received by whichever pipeline reads it first and is then removed from the queue, so the other pipeline never sees it.
Neither pipeline reports an error, and each one receives only part of the data.
:::

This applies to every input that takes messages off a queue or subscription, whether the two pipelines share one input or use two separate inputs configured with the same source:

| Source | Shared by pipelines that use the same… |
| ------ | -------------------------------------- |
| Amazon SQS (SQS, SQS S3, SQS S3 CloudTrail, SQS S3 GuardDuty, Security Hub Findings, Falcon Data Replicator) | Queue URL |
| Azure Queue Storage | Storage account and queue |
| Google Cloud Pub/Sub | Subscription |
| Kafka and Confluent Messages | Topic and consumer group |

In a test with two pipelines on one SQS queue, 30 S3 objects were divided 16 to one pipeline and 14 to the other.

To send the same queued data to more than one destination, use one of these patterns instead:

- **One pipeline, several outputs.**
  Keep a single pipeline reading from the queue and add an output, with its own edge and any transforms or conditions, for each destination.
  This is the simplest option and reads each message only once.
- **One queue per pipeline.**
  If the data must go through separate pipelines, give each pipeline its own queue.
  For S3 event notifications, send the bucket's notifications to an Amazon SNS topic and subscribe one SQS queue per pipeline to that topic, so every queue receives every notification.
  See [Fanout to Amazon SQS queues](https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html) in the AWS documentation.
  For Azure Queue Storage fed by Event Grid, create one Event Grid subscription per queue.
  For Pub/Sub, create one subscription per pipeline on the same topic; every subscription receives every message.
  For Kafka or Confluent, give each pipeline its own consumer group.

A queue that a vendor manages for you, such as the CrowdStrike Falcon Data Replicator queue, cannot be fanned out this way, so read it from a single pipeline.

## Azure Event Hubs

Reading from an Event Hub does not remove events, and each pipeline tracks its own position in every partition.
Pipelines that use the same [Azure Event Hubs](../inputs/microsoft/azure-event-hubs.mdx) input, even with the same consumer group, therefore each receive every event.
Two limits apply:

- **Azure allows at most five readers per partition in one consumer group.**
  Each pipeline opens one reader per partition, so a sixth pipeline on the same consumer group (or fewer, if other applications also read from it) cannot connect.
  Give each pipeline its own consumer group to avoid the limit.
- **A pipeline added later only reaches back as far as Lookback Duration.**
  On its first run a pipeline starts reading at the current time minus **Lookback Duration**, so it does not receive older events that an earlier pipeline already read, even if the Event Hub still retains them.

## Polling cloud storage

Each pipeline that uses an [S3](../inputs/aws/s3.mdx), [Azure Blob Storage](../inputs/microsoft/microsoft-azure-blob-storage.mdx), or [Google Cloud Storage](../inputs/google/google-cloud-storage.mdx) input lists and reads the bucket or container on its own, so every pipeline receives every object.
That also means every pipeline makes its own list and read requests, so N pipelines on one bucket cost N times the storage requests of one.
The rest of this section uses S3 as the example; the other storage inputs behave the same way.

Each pipeline also keeps its own checkpoint, which starts at the input's **Backfill Start Time**.
When **Backfill Start Time** is blank, the checkpoint starts at the moment the pipeline first runs.
A pipeline added later therefore never receives the objects written before it started, even though it uses the same input as an earlier pipeline:

1. Pipeline A starts, and 10 objects are written to the bucket. Pipeline A receives all 10.
2. Pipeline B is created with the same input, and 5 more objects are written. Pipeline A receives the 5 new objects, for 15 in total. Pipeline B receives only the 5 written after it started.

To give a later pipeline the same history, set **Backfill Start Time** to a date that covers the objects you need before that pipeline first runs.
Changing **Backfill Start Time** after a pipeline has already run has no effect on that pipeline, because it resumes from its saved checkpoint.

## Syslog

A [Syslog](../inputs/monad/monad-syslog.mdx) connection is routed by the first of these that identifies a pipeline:

1. **The hostname you connect to (SNI).** The connection goes to that one pipeline.
2. **A client certificate whose Subject `serialNumber` is a pipeline ID.** The connection goes to that one pipeline.
3. **A registered certificate fingerprint.** The connection goes to **every** pipeline that uses the input the fingerprint is registered on, and each pipeline receives an identical copy of every message.

Registering a fingerprint on an input that several pipelines use is therefore the way to send one syslog stream to multiple pipelines.
Keep the following in mind when you rely on it:

- **A disabled pipeline drops out silently.**
  While one of the pipelines is disabled, the sender gets no error and the other pipelines keep receiving data.
  Syslog is not replayed, so when the pipeline is re-enabled it does not receive the messages it missed, and the pipelines stay permanently out of step.
- **The certificate wins over the hostname.**
  A certificate carrying a pipeline ID in Subject `serialNumber` routes to that pipeline when the client does not send SNI, whatever hostname it was configured with.
  A device holding a certificate with another pipeline's ID delivers into that other pipeline.

## HTTP

Every pipeline has its own [HTTP](../inputs/monad/monad-http.mdx) ingest endpoint, even when several pipelines use the same HTTP input.
Data sent to one pipeline's endpoint reaches only that pipeline, so to deliver the same data to two pipelines, send it to both endpoints.

## Rate limits and API quotas

A rate limit configured on an input applies separately to each pipeline that uses it.
An input limited to 1 request per second and attached to three pipelines can make up to 3 requests per second against the source API.
When several pipelines share an input, divide the source's quota among them, as described in [API Rate Limiting](./rate-limiting.mdx).