The Best Way to a solution is to “Split the Problem”. This Architecture helps in designing the Data Pipeline with the various requirements of either the Batch Processing System or Stream Processing System. Secured data flow can be viewed under Big Data Architecture in 6 layers that ensures data security.
Data Ingestion Layer
This layer is the first step for the data coming from variable sources to start its journey. Data here is prioritized and categorized which makes data flow smoothly in further layers.
Data Collector Layer
In this Layer, more focus is on the transportation of data from ingestion layer to rest of data pipeline. It is the Layer, where components are decoupled so that analytic capabilities may begin.
Data Processing Layer
In this primary layer, the focus is to specialize in the data pipeline processing system, or we can say the data we have collected in the previous layer is to be processed in this layer. Here we do some magic with the data to route them to a different destination, classify the data flow and it’s the first point where the analytic may take place.
Data Storage Layer
Storage becomes a challenge when the size of the data you are dealing with, becomes large. Several possible solutions can rescue from such problems. Finding a storage solution is very much important when the size of your data becomes large.
Data Query Layer
Main data analytics processing is done here. The primary focus is to gather the data value so that they are made to be more helpful for the next layer.
Data Visualization Layer
Visualization Layer is the most important layer and this is phase where the users can have the look and feel and important of Data. This provides the view for the end users of the actual VALUE of the Data synthesized.