syntioinc/dataphos-persistor-core repository overview

The main component of Dataphos Persistor. Subscribes to a topic and sends messages to persistent storage. It is the only non-optional component.

Supports receiving messages from Google Pub-Sub, Azure Service Bus or Apache Kafka. The available storage options are GCS, ABS, and MongoDB.

Enables folder structure based on the publish time of a message (coarse or fine-grained, down to the hour), also supports the ability of utilizing arbitrary metadata.

More info: https://docs.dataphos.com/⁠