Welcome to Post 2 in a series of posts on Data Science – Recommendation Engines (RE).
In this post, we discuss on Data Profiling and Data Processing (Data Pipeline) aspects of domain objects.
In Post 1, we discussed on Users -> Items -> Users & Items interactions, which I consider as core enterprise or business applications domain objects.
This post assumes below steps are applied to the “User(s)” and “Item(s)” datasets.
Below are the steps, I think, one should follow during Data Profiling and Data Processing, before starting the design and development of RE models.
Step 1: Data Profiling, Features Identification and Establishing Discrete or Continuous or Categorical Step 2: Normalization and Standardization of Features into Numbers OR Boolean OR Ranges Step 3: Establish a data relationship between Users & Items dataset
Finally Plan, Design and Develop Data Pipelines that meets all the above steps.
Step 1: Data Profiling, Features Identification and Establishing Discrete or Continuous or Categorical
Data Profiling provides a good understanding of Users / Items / Users & Items interactions data like
1. How many features/columns available in the overall dataset – get the number like location, age, likes, preferences, other use connections
2. For each feature establish whether it is a Discrete Value OR Continuous Value or Categorical Value
a. Discrete – countable, separate, distinct
b. Continuous – not countable
c. Categorical – finite discrete
3. Minimum, Maximum, Average values for the identified features
In my experience, soon after a Data Ingestion workflow is executed we run Data Profiling workflows using Apache Pig or Apache Spark or ANSI SQL.
Above profiling workflows builds statistics for the newly ingested data such as Min, Max, Average and stored them either in MongoDB or AWS DynamoDB or any other NoSQL databases.
Step 2: Normalization and Standardization of Features into Numbers OR Boolean OR Ranges
1. For each identified feature value, determine how you want to represent it as 0/1 ’s OR True / False OR Range of values
2. Identify missing values and transform them also into the above representations
3. Overall, in this step, we do normalization of source data
Normalization is the process of rescaling the observations to a notionally common value
There are 3 types of normalization
a. Min-Max Normalization
b. Z-Score Normalization
c. Decimal Scaling
Step 3: Establish a data relationship between Users & Items dataset
1. From the overall feature set, identify how User preferences can be connected with Item features
a. Example: Users liking a hotel (an item) can be considered as a preference
Plan, Design and Develop Data Pipelines
While all the above steps are theoretical and important, below are the activities a data engineer should perform:
1. Document all the transformations (data pipeline) required to build new Features / Columns
2. Develop data pipeline code which does required transformations of Features / Columns into Numbers or Boolean or Ranges
3. Store the transformed and final representation of data (columnar Parquet or JSON or Avro) that is ready for building Recommendation Engine Models
Here are the links to other posts Part 01 Part 03
- The AI Blueprint: Decoding “Latent Space” in Latent Diffusion Models (LDMs)
- NLP Made Simple: The 12 Core Tasks Explained in One Table
- AI LLM Learnings – Understanding Words, Sentences, and Tokens – Using a Real Example from AWS EC2 Documentation
- Agentic AI Design Patterns for DevOps in Cloud-Native Kubernetes Environments
- AI – Building a Real-Time FOREX Position Risk Manager with VLLM, AWS, and Bedrock
- Machine Learning Experiences: Enhancing Synthetic Trade Data for FOREX Risk Modeling
- Machine Learning Mastery – Understanding Ensemble Methods And Their Benefits
- Chief Architect Decision Making – Matrix for AI in Forex Risk Ops
- Chief Architect AI Lens – Enterprise AI Blueprint for Forex Position Risk Operations