Transwarp - Transwarp Operating System
Transwarp Operating System (TOS)
Overview
The soon to be released TOS is a cloud operating system tailored for big data applications. Based on Docker and Kubernetes, TOS allows you to deploy TDH with one click. Big data applications can now embrace cloud services with the help of fine-grained resource distribution and preemptive resource allocation by priority.

The demand for an enterprise data hub to drive their businesses is ever growing for future companies. A data hub needs to provide the following:

1. Elastic resource sharing for higher utilization

2. Isolation to ensure service quality and security

The newly developed TOS satisfies all the above needs of an enterprise data hub. It provides one-click deployment, scale up and scale down of TDH while allowing other services to share the cluster with big data services, which makes better utilization of resources. While ensuring real-time tasks, the innovative preemptive resource allocation model of TOS increases resource occupation when the cluster is idle, which allows batch jobs and real-time tasks to share computation resources without interfering each other.

In addition, TDH over TOS introduces micro service architecture, which significantly alleviates the influence of customer deployment environment on the stability of TDH, increases the availability of the deployment and allows the user to upgrade and fix the system without downtime.

Components of Transwarp Operating System

Transwarp Operating System is composed of the following components

  • Ring 0: Docker/Container
  • Ring 1: Resource scheduler
  • Ring 2: Built-in system services
  • Ring 3: Central service repository (docker images)
HDFS YARN Incep
tor
Stream Elastic
Search
Hyper
base
Pig Oozie Flume Sqoop Postgr
esSQL
Redis

Service Repository

Technical Advantages
TOS provides one-click deployment, scale up and scale down of TDH while allowing other services to share the cluster with big data services, which makes better utilization of resources. While ensuring real-time tasks, the innovative preemptive resource allocation model of TOS increases resource occupation when the cluster is idle, which allows batch jobs and real-time tasks to share computation resources without interfering each other.

1. Austomatic Deployment Play

With just one click, the user can install and deploy her cluster via Web, REST API or CLI. All products and services will be installed automatically according to their dependencies.

HDFS YARN Incep
tor
Stream Elastic
Search
Hyper
base
Flume Sqoop Oozie Pig Postgr
esSQL
Redis
HDFS YARN Incep
tor
Stream Elastic
Search
Hyper
base
Flume Sqoop Oozie Pig Postgr
esSQL
Redis

Service Repository

HDFS YARN+ Map/Re
duce
Inceptor Stream Elastic
Search
Hyperbase Flume Sqoop Oozie Pig

2. Comprehensive Multi-Tenant Support Play

Transwarp Operating System has realized better isolation of the CPU, memory, disk and network by rewriting the resource management framework. The following is a comparison of TOS, YARN and Kubernetes.

  Resource Size Isolation Level Dependency Generality
YARN CPU/MEM Process level, not precise Depends on some HDFS Support a few computing engines
Kubernetes CPU/MEM Container Independent of Hadoop Support general Linux load
TOS CPU/MEM, DISK, NETWORK Container + Quota + VLAN Independent of Hadoop Support big data and general applications
HDFS1 Inceptor1 Data
Warehouse Apps
Inceptor2 Datamarts Analysis Mining HDFS2 Hyperbase HBase Online Query Apps Stream Real-time LBS Apps

3. Automatic Capacity Adjustment Play

TOS provides one-click deployment, scale up and scale down of TDH while allowing other services to share the cluster with big data services, which makes better utilization of resources. While ensuring real-time tasks, the innovative preemptive resource allocation model of TOS increases resource occupation when the cluster is idle, which allows batch jobs and real-time tasks to share computation resources without interfering each other.

HDFS1 Inceptor1 Data
Warehouse Apps

4. Automatic Repair Play

The Replicator module of TOS examines the size of the cluster so that when a service experiences problems, the Replicator will start a new service instance allowing TOS to automatically repair itself.

Stream Real-time LBS Apps Stream