Data Platform Considerations – Technology Focused

Data & Analytics, Uncategorized

Solution Architects in a large enterprise while creating a Data Platform solution architecture should consider following factors besides business factors.

  1. Data and Compute Separation 
  2. Fixed technology stack with governance for new stack
  3. Centralized repositories for domain data, metadata, information assets
  4. Data Platform Characteristics supported

1. Data and Compute Separation

  • Architecture should have a logical (and physical) separation of Data and Compute resources
  • Data Virtualization capability to query multiple data management platforms (HDFS, S3, GCP, Cacti, Cassandra, HBase etc)
  • Containerization of compute resources based on simple and scalable architecture characteristics
  • Support of different file formats of data storage
  • Centralized Datalake and Enterprise Data Warehouses with open file formats and open data architecture

2. Fixed Technology Stack with Governance for New

  • Spark As a Service including Enhanced Spark Runtime, multi-versions Spark runtime
  • Scheduling and Cron support that are enterprise approved tools (Autosys example)
  • Workflows that support Transactional Upserts & Deletes ( ACID compliance )
  • Batch and Streaming technologies
  • Technologies that support updates, deletes and upserts
  • OLAP use cases Data management technology (Teradata, Greenplumn, HDFS, AWS S3)

3. Centralized Repositories for Domain Data, Metadata, Information Assets

A centralized repository from Data Platform Administration perspective:

  • Data Pipelines Deployment Repository with version control
  • Data Pipelines supporting server use-cases
  • Business and associated Data Applications On-Boarding and their management
  • Data Catalog for Business Data and their supporting Services discovery, routing a specific data asset or information assets on their metrics (data quality metrics, ingestion / data transformation  processing, source data latest on which such transformations or processing happened to build the data asset or information asset)

4. Data Platform Characteristics Supported

Characteristics in technology and data processing services that are

  • Scalable
  • On-Demand
  • Elastic 

Example 1: Spark As a Service, Airflow As a Service, IceBerg As a Service, Authentication As a Service, Metadata Management As a Service, Auditing As a Service – provisioning of these services should have above characteristics

Example 2: Platform should support daily 100 million Jobs, 2 to 3 Million Executors per dat, 4 Petrabyte of RAM across various data centers, 

Post Views: 506