Deploying on a State Data Center

[ChakshuGautam]

Hey everyone, wanted to figure out if Obsrv has scripts to deploy on a State Data Center (SDC). The challenges are the following

1. No k8s available.
2. SSDs are not available.
3. Limited or NO support for Internet.

Thanks

cc:- @suresh12 @surendrasinghs

[rhwarrier]

Not clear what the intent is - what is the need to be addressed?
'Deploying Obsrv' for what? Enabling reporting of some sort? From what data?
A little more context would help.

@anandp504 @SanthoshVasabhaktula

[ChakshuGautam]

Hey, @rhwarrier would you please explain how 'Deploying Obsrv' on an infra is coupled with a use case? I would want to understand since that affects the choice of telemetry backend as well.

If the deployments are coupled with use cases, can you share docs on how?

[SanthoshVasabhaktula]

@ChakshuGautam - We have installation scripts for AWS and Azure. GCP is work in progress. Automated scripts for a data center deployment is open for community contribution.

[surendrasinghs]

We would like to use it for de-duplicating records (match 1 against ~3Cr), generate aggregate reporting metrics, and so on.

[SanthoshVasabhaktula]

@surendrasinghs @ChakshuGautam - Obsrv would work on a data center as well. The developers run every component of Obsrv locally on their laptops as well in a K8s cluster. However automated provision and deployment scripts are not available and needs contribution from the community.

[rhwarrier]

Aggregate metrics for reporting is something that's possible, of course. Assuming the scale justifies usage of an infra based tool such as Obsrv. You'll have to take a call on that.
De-duping is better served using an off-the-shelf tool. Obsrv is infra, it'll be much simpler to use some other existing tool for operations such as this.

[ChakshuGautam]

Hey, @rhwarrier confirming this once. Please forgive if some of the assumptions are wrong, I am trying to leverage as much as I can from the existing modules/constructs that Obsrv has

1. Collecting facts (of transactions events) - Can be done through Obsrv as an infra. As Santhosh said, it can be deployed using docker-compose on a Swarm in a scalable manner without k8s (please confirm if this is the case).
2. De-duping - Should be done using some other tool (probably spark) that would query facts for say the last year and do whatever with that. I see that the Obsrv leverages spark extensively for report generation - which is a similar use case.

[SanthoshVasabhaktula]

@ChakshuGautam - Obsrv can be deployed on K8s only not docker swarm. We don't have scripts for it. You can use the docker images and deploy on a swarm. Deduping is done via Flink during the data ingestion phase.

[ChakshuGautam]

Thanks @SanthoshVasabhaktula for clarifying this. Have some follow-up questions on de-duping,

1. What is the average cost of getting a single event from the Obsrv infra? Assuming that

N=Total Number of events and
P=Total number of partitions for those events on storage for those events.
... It could totally not depend on the above two as well and do an O(1) lookup some other way.

what would be a rough big-O for Sunbird Obsrv for this use case?

Alternatively, I saw that this is being done using Redis by matching keys. For one of our use cases, we would want to configure Redis to store only a certain type of events, will this be possible?

What happens if the key is not in the cache?

2. We are planning to use this as a transaction system (@surendrasinghs please chime in here). I am assuming Obsrv is a telemetry store and there are no ACID guarantees here but I am assuming Obsrv will not miss anything that was delivered to the API.

Please let me know if I need to create separate tickets for all of these.

[SanthoshVasabhaktula]

@ChakshuGautam -Somehow I have missed answering this.

1. What do you mean by average cost? Are you looking for the infra running cost or avg processing time? - This is completely dependent on how the infra is configured, is it optimised for speed or cost
2. The purpose of redis for two things
   2.1 Dedup the events for a certain ttl. Here the dedup job checks whether the event "mid" is in redis. If the key is in redis the event is marked as duplicate else the key is added into redis with a ttl
   2.2 To cache master data for data denormalisation. Here the denormaliser job deforms the event with master data if the id (for ex: userId, contentId etc) is present in the cache
3. While Obsrv is built as zero data loss system, it is not built to serve as a transactional system. I would recommend using a traditional columnar database like cassandra for transactional storage needs.

[rhwarrier]

That sounds about right, yes.
Anything else to add, @SanthoshVasabhaktula ?