New Real-Time Stream Processing Platform Powers Live Data Apps

11 May 2018
in: Company News

Release of dA Platform with open source Apache Flink enables enterprises to process, analyze and react to massive quantities of data instantaneously. It moves a little quicker than Hadoop and other systems.

Data Artisans, the Berlin, Germany and San Francisco-based maker of Flink, has announced the general availability of dA Platform, which it describes as the industry’s first “turnkey” stream-processing platform to enable enterprises to obtain insights from data in milliseconds.

The new release productionizes stream processing and enables companies to provide live data applications as a centralized enterprise service. dA Platform reduces the manpower, cost and effort required to deploy stream-processing applications in production, and provides a reliable and high-impact stream processing platform across an organization.

Stream processing is a programming paradigm, equivalent to dataflow programming, event stream processing and reactive programming that allows some applications to more easily exploit a limited form of parallel processing.

Data Artisans already makes Cloud Dataflow runner for Flink, which allows any Dataflow program to execute on a Flink cluster located in the cloud or installed on-premise.

Cloud Dataflow Originally from Google

Cloud Dataflow was originally released as a service on Google’s Cloud Platform four years ago. Thereafter, Google released a Cloud Dataflow Software Development Kit (SDK) for developers looking to port the programming model to other processing engines. In January 2015, Google and Cloudera announced the availability of Dataflow on Cloudera’s popular Apache Spark platform.

Flink is a 5-year-old, top-level project from the Apache Software Foundation that offers a distributed processing engine for running batch and steam-processing applications. Data Artisans describes it as an alternative to Hadoop’s MapReduce component that is capable of working independently of the Hadoop ecosystem.

Following an early-access program in which the platform was extensively tested across large-scale environments, Data Artisans told eWEEK that dA Platform is enterprise-ready. A trial version is now available for download here.

Analysts have projected that the streaming analytics market will reach $15.9 billion by 2022, because companies across industries are transitioning from a product-centric business model to one based on relationships and a services-centric model.

Streaming analytics for various types of use cases is ramping up. For example:

auto manufacturers are introducing new car-sharing and ride-sharing services as car ownership shifts to a usage-based model;
consumer banks are creating new messaging applications for communicating with customers in real time to provide more seamless management of their personal finances; and
insurance companies are offering dynamically priced insurance products tailored for customers based on their usage data.

Stateful stream processing has emerged as the technological standard to enable this transformation, Data Artisans said.

dA Platform, which includes open source Apache Flink, is the first toolset purpose built for stateful stream processing for enterprise engineering organizations, unifying different components to provide a seamless experience with deployment and operations. Flink processes data in real time, is designed for unbounded datasets and has become the stream processing engine of choice for streaming data applications.

Highlights of dA Platform

dA Platform, including open source Apache Flink, features the new Application Manager, which streamlines the process of deploying and maintaining real-time streaming applications with Flink in production, greatly reducing the time to market and personnel required for businesses to realize value from streaming. It integrates with an organization’s existing streaming data sources, developer workflows, service deployment architecture, and logging and metrics infrastructure to be the nexus of all streaming data processing applications within the organization.

DA Platform offers the following:

orchestrates the lifecycle of Flink applications through development cycles;
integrates with logging and metrics systems to help engineers during debugging and development;
deploys streaming applications on Kubernetes for dynamic resource allocation;
automates common operations like redeploying an updated Flink application or initiating a savepoint; and
maintains records of all user actions and automatic processes for later auditing

Global companies such as Alibaba, Capital One, Cisco, ING, Lyft, Netflix and Uber use Flink for large-scale stateful streaming applications, including real-time analytics, fraud/anomaly/threat detection, machine learning, and search and content ranking, the company said.

Pricing and Availability

Data Artisans is offering a free trial sandbox that enables users to start testing dA Platform in minutes. A trial version that users can install on their own Kubernetes infrastructure is also available. The enterprise-ready dA Platform is available now through a one-year subscription, with usage-based pricing (number of cores). You can download the free trial here.