I am working on an approach to creating a data lake using Cloudera. My question is for an approach to be taken for the Curated/Gold layer.
RAW layer - All data from the source will be available in raw format and partitioned as per the data extraction key.
Gold layer / Curated Layer / Optimized layer - Here in this layer, the plan is after reading the data from RAW layer will do simple transformation on column values, transformation on file format changes to Parquet (partition, compression, enrichment) and then write as a parquet file using Apache Spark. I am not planning to create a hive table in this spark job or use a hive query to create a parquet table. Instead will create a separate impala query/hive query to read data from this file as per need.Idea is that there should be flexibility to query the data using any component - Impala/Hive/Presto etc. Incase i create a parquet file using hive script and if need to read this data from Impala, then the Impala query compilation then hive query compilation and other required processing could be overhead.
Data serving layer: Here will create Hive HQL/impala scripts to read data from Gold/Optimized layer to read data.
What is your opinion in terms of storing data in gold layer- as a partitioned table using a hive or store as a partitioned parquet file and then read using any component?
