AWS S3 as an Interface Between Independent Event Systems

The integration problem is not always technical

When two independently operated systems need to exchange a large volume of events, the default response is often to introduce a queue, event bus, or streaming platform. That may be justified, but it also introduces shared infrastructure and additional coordination around platform ownership, access, schemas, and operational responsibilities.

In many organisations, the hardest parts of integrating independently operated services are not technical. Competing priorities, different KPIs, ownership boundaries, constrained capacity, and busy roadmaps all create friction.

In these scenarios, there is significant value in choosing the simplest integration point that allows teams to cooperate while minimising mutual dependencies, both technical and operational.

The core pattern: immutable files plus a consumption ledger

The pattern is straightforward: the event producer writes immutable batches of events to a defined S3 location, and the event consumer discovers and processes new files independently.

S3 holds the durable event history, while a small consumption ledger, typically a database table, records which files have been discovered, claimed, completed, or failed. Together, these components provide a replayable integration boundary without pretending that S3 itself offers acknowledgements, retries, or consumer state.

When no suitable shared messaging platform already exists, publishing immutable files to S3 can require substantially less cross-team coordination than introducing and operating shared streaming infrastructure.

Flow A: incoming events
Event producer -> S3 incoming/ -> consumer ledger -> event consumer

Flow B: result events
Event consumer -> S3 results/ -> producer ledger -> original producer

A typical implementation compares the files present in S3 with the files recorded in the ledger, inserts unseen objects as pending work, and lets workers claim and process them atomically. The exact implementation varies, but the reusable design is the separation between immutable source data, explicit consumption state, and idempotent downstream effects.

The interface can also work in both directions

The same pattern can support asynchronous communication in both directions. After processing an input file, the event consumer can publish its own immutable result events to a separate S3 prefix for the original producer to consume.

These output events may represent simple acknowledgements, detailed processing results, or transformed business data. For example, an event producer may publish batches of ecommerce products, while the event consumer processes them and publishes back whether each product is suitable for international sale, together with any restrictions or reasons.

It is useful to model this as two independent unidirectional flows rather than one bidirectional queue. Each direction should have its own schema, prefix, consumption ledger, retry policy, and ownership boundary.

Why S3 can be the simplest viable solution

For batch-oriented workloads, S3 can deliver most of the properties the integration actually needs with very little shared infrastructure. Storage is inexpensive, producers and consumers can remain available independently, and throughput can grow by partitioning files and adding workers.

The same retained data supports replay, backfills, auditing, and future consumers without a separate archival path. When delivery latency of seconds or minutes is acceptable, this can be materially easier to build and operate than a richer messaging platform.

Storage costs can be controlled through S3 Lifecycle rules that expire data after a defined retention period or transition older objects into less expensive storage classes.

A queryable event history with Athena

The files in S3 are not only transport artefacts; they can also become a directly queryable event archive. By defining an Athena external table over the bucket or prefix, engineers can inspect historical events using standard SQL without first loading them into another database.

This is useful for debugging, audits, volume analysis, and validating producer output. The benefit depends on disciplined schemas and partitioning: JSON Lines is easy to adopt, while formats such as Parquet reduce scan costs for larger or more frequent analytical queries.

Retention policy should distinguish between data that must remain immediately queryable and data kept only for long-term recovery or compliance. Objects moved into archival storage classes may need to be restored before they can be queried, so cold storage reduces cost at the expense of access time and operational convenience.

The complexity S3 does not remove

S3 simplifies the shared infrastructure, but it does not eliminate the hard parts of reliable event processing. The consumer must still handle duplicate delivery, idempotent writes, atomic work claiming, retries, poison files, schema evolution, partial failures, and backlog monitoring.

These responsibilities are manageable when the workflow is simple, but they should be counted honestly. The pattern stops being lightweight when the application begins rebuilding a large portion of the delivery, routing, and operational semantics already provided by a mature broker.

The key test is whether the ledger, retry model, and monitoring remain materially simpler than adopting the messaging platform they replace.

The same is true for bidirectional workflows: acknowledgement events and business-result events still need their own schemas, correlation IDs, and idempotent consumption logic.

Polling, notifications, or both

A consumer can discover files by periodically listing S3, by reacting to object-created notifications, or by combining both approaches. Polling is simple and naturally reconciles missed work, but it introduces discovery latency and repeated listing costs.

Notifications provide faster delivery, yet remain at-least-once and add another component such as SQS or EventBridge. A practical compromise is to use notifications for prompt processing while retaining periodic S3 reconciliation as the correctness mechanism.

Trade-offs against structured messaging systems

S3 wins on durable retention, large batch handling, replay, low storage cost, and direct inspection of historical events. Queues and streams win on low latency, per-event acknowledgements, ordering, routing, retries, and consumer coordination.

The relevant comparison is therefore not whether S3 can imitate Kafka, SQS, or EventBridge, but whether the application needs the richer guarantees those systems provide. Choosing S3 is a deliberate trade: fewer shared infrastructure components in exchange for simpler delivery semantics and more responsibility in consumer code.

Criterion	S3 files	Queue or event bus	Event stream
Typical latency	Seconds to minutes	Milliseconds to seconds	Low-latency continuous delivery
Payload model	Files and batches	Individual messages or events	Continuous ordered records
Retention and replay	Natural and inexpensive	Varies by service and configuration	Strong within the configured retention window
Acknowledgements and retries	Implemented by the consumer	Usually managed per message	Offsets plus application-level handling
Ordering	Custom; not inherent	Limited or service-specific	Typically guaranteed within a partition
Historical queryability	Direct through Athena	Usually requires export or a separate sink	Usually requires a separate analytical sink
Operational model	Low shared infrastructure; more consumer logic	Managed delivery semantics	Richer semantics and greater platform complexity

When to choose this pattern

This pattern is a strong fit when:

• events are naturally produced in files or batches;
• delivery can tolerate seconds or minutes of latency;
• retained history has operational or compliance value;
• replay and backfills are likely requirements;
• engineers benefit from querying historical events through Athena;
• future consumers may need to read the same source data independently;
• downstream writes can be made idempotent;
• the workflow benefits from publishing acknowledgements or transformed results back through a separate S3 output prefix;
• the ledger, retry logic, and monitoring remain simpler than operating a broader messaging platform.

When not to choose it

A queue, stream, or event bus is usually the better choice when:

• events must arrive in near real time;
• strict or partition-level ordering is important;
• each event needs independent acknowledgement or retry;
• routing, filtering, or fan-out are core requirements;
• duplicate delivery cannot be handled safely;
• individual record failures require precise retry control;
• the organisation already operates a suitable messaging platform, making a second delivery model more complex rather than simpler.

Conclusion: choose the least structured tool that meets the guarantees

S3 should not be presented as a drop-in replacement for a message broker. It is a different integration model that works well when durability, batch throughput, replay, queryability, low cost, and loose coupling matter more than sophisticated delivery semantics.

The architectural decision should start from the guarantees the system genuinely needs. When those guarantees are modest, immutable files and a small ledger can be the simplest solution. When they are not, a structured messaging system earns its additional complexity.