Near real-time ROAS (Return on Ad Spend) engine processing terabyte-scale user acquisition data. Built on AWS with Apache Spark, handling GCS-to-Iceberg ETL pipelines.
The problem
The platform's user acquisition data pipeline was processing terabytes of gaming data with significant performance degradation and unexpected cloud egress costs. The root cause was Spark lazy evaluation behaviour and improper DataFrame caching, which was not surfaced by existing monitoring.
Building it
Bayseian audited the full Spark pipeline architecture on AWS, identified the lazy evaluation and caching issues, and rebuilt the GCS-to-Iceberg ETL with correct execution plans, partition strategies, and caching policies.
What changed
Pipeline performance recovered significantly with reduced cloud egress costs. Near real-time ROAS attribution now processes at the required throughput for production user acquisition operations.
