...
- Do not block. Use non-blocking libraries. Do not use block* Reactor calls inside the core of the application.
- Pay attention to memory usage.
- Do not decode the payload - it can be of a considerable size. The goal is to direct the event into a proper Kafka topic. The routing logic should be based only on VES Common Header parameters.
- All application logic should be defined in hv-collector-core module and tested on a component level by tests defined in hv-collector-ct. The core module should have a clean interface (defined in boundary package: api and adapters).
- Use Either functional data type when designing fail-cases inside the main Flux. Using exceptions is a bit like using goto + it adds some performance penalty: collecting stack trace might be costly but we do not usually need it in such cases. RuntimeExceptions should be treated as application bugs and fixed.
Compatibility
...
aspects (VES-JSON
...
)
- VES-HV has been designed as a high-volume variant of the existing VES(JSON) collector, and not a completely new collector
- VES-HV follows the VES-JSON schema - as much as possible
- It uses a PROTO representation of the VES Common Header
- The PROTO files tend to use most encoding effective types defined by GPB to cover Common Header fields.
- It makes routing decisions based mostly on the content of the "Domain" parameter
- It allows to embed Payload of different types (by default hvMeas domain is included)
- VES-HV publishes events on DMaaP-Kafka bus, using native Kafka Interfaces
- Analytics applications impacts
- An analytics application operating on high-volume data needs to be prepared to read directly from Kafka
- An analytics application need to operate on GPB encoded data in order to benefit from GPB encoding efficiencies
- It is assumed, that due to the nature of high volume data, there would have to be dedicated applications provided, able to operate on such volumes of data.
...