Transwarp Stream Processing Framework

Transwarp Stream real time stream processing endgine delivers great stream computing capability, supports complex application logic. Information from production systems enters computing clusters via real time message queues. Then it is processed via pipelines inside clusters to complete complex service computations including data conversion, characteristics extraction, strategy inspection and analysis warning. The results of such computations are eventually output to storage clusters such as Hyperbase to create real time warning and display pages. The system is highly scalable and highly fault-tolerant. It has large throughput and low latency. Transwarp Stream has been successfully applied to real-time warning, risk control, online statistics and mining based on real-time data (such as sensor data).

Transwarp Stream Architecture
Transwarp Stream Functionalities
Functionalities Description
Kafka Distributed message queue with low latency and high throughput which supports the announce/subscription model, supporting both online and offline message distrbution systems. Transwarp has realized a strategy for access control management such that only authorized users can perform read and write on Kafka clusters, avoiding data leaks and misuses.
Stream Processing Engine A stream computation engine with high throughput. It is highly expressive for stream computation. It supports complex computation logic on stream data including real-time event detection and machine learning.
Interactive discovery analysis Supports transforming real-time data streams to columnar data to be stored in Holodesk. In addtion, it can perform ad-hoc SQL-analysis or R-data-mining via Inceptor.
Machine Learning on Streams Supports real-time machine learning in the process of computation. For example, in cluster analysis, it can adjust the cluster center in real time, and in classification, it can update classification models in real time and compare them to data streams.