Una plataforma de Apache Pulsar como servicio con pricing transparente, herramientas de debugging simplificadas y onboarding guiado para reducir la curva de aprendizaje

Background: At RudderStack, I had been successfully using Postgres for the event streaming use case, scaled to 100k events&#x2F;sec (note: there were good reasons to choose Postgres over Kafka). Nevertheless, we continue to further explore opportunities to optimize. So I and my team started experimenting with Pulsar (only for the parts of our system - data ingestion specifically). We experimented with Apache Pulsar for ingesting data vs having dedicated Postgres databases per customer (one customer can have 1+ Postgres databases, they would be all master nodes with no ability to share data which would need to be manually migrated each time a scaling operation happens).<p>Now that it&#x27;s been quite some time using Pulsar, I feel that I can share some notes about my experience in replacing postgres-based streaming solutions with Pulsar and hopefully learn from your opinions&#x2F;insights.<p>----<p>What I liked about Pulsar:<p>1. Tenant isolation is solid, auto load balancing works well: We haven&#x27;t experienced so far a chatty tenant affecting others. We use the same cluster to ingest the data of all our customers (per region, one in US, one in EU). MultiTenancy along with cluster auto-scaling allowed us to contain costs.<p>2. No more single points of failure (data replicated across bookies): Data is replicated in at least two bookies now. This made us a lot more reliable when it comes to data loss.<p>3. Maintenance is easier: No single master constraint anymore, this simplified a lot of the infra maintenance (imagine having to move a Postgres pod into a different EC2 node, it could lead to downtime).<p>----<p>What&#x27;s painful about Pulsar:<p>1. StreamNative licensing costs were significant<p>2. Network costs considerably increased with multi-AZ + replication<p>3. Learning curve was steeper than expected, also it was more complex to debug<p>----<p>Would love to hear your experience with Postgres&#x2F;Pulsar, any opinions or insights on the approach&#x2F;challenges. I hope this dialogue helps others in the community, feel free to ask me anything.