Team members new to AWS need to learn and master the power of AWS Cloudwatch, especially its events monitoring, metrics monitoring, alarms, and notifications.
In this post, I share one of my experiences and the need for applying “IsolationForest Machine Learning” to AWS CloudWatch events monitoring.
When a Dot.Net microservice was deployed in AWS EC2 with a MongoDB database, on an important demo day, the Dot.Net service was shutting down consistently, and developers could not identify the root cause. Though the Cloud Watch is installed on the AWS EC2, the lack of AWS knowledge and skill by the developers prevented identifying the root cause.
Revealing the excitement early on the root cause. The issue was with the MongoDB FIND query, the Dot.Net service trying to fetch 1 million rows from the DB.
We were not prepared for an outlier scenario where a specific use case generated more than 1 million documents in a MongoDB collection. Even though the MongoDB FIND Query applied MongoDB query on indexed columns, the outlier scenario made the service shut down consistently. Until the outlier data is shredded out of the MongoDB collection,.
Post-issue, one of the techniques we decided to deploy is monitoring various operational sources, and CloudWatch is one of the sources. Through the “Isolation Forest Machine Learning” technique, we could identify the memory usage outlier.
The IsolationForest is a machine-learning algorithm used for anomaly detection. It is particularly effective for detecting outliers, though, in high-dimensional datasets. In this use case, we might not have high dimensions, but we still applied the technique.
Below is a Python code example for AWS CloudWatch metrics monitoring using IsolationForest.
import boto3
import numpy as np
from sklearn.ensemble import IsolationForest
cloudwatch = boto3.client('cloudwatch')
# Function to fetch CloudWatch metrics data
def get_cloudwatch_metrics(metric_name, namespace, start_time, end_time):
response = cloudwatch.get_metric_data(
MetricDataQueries=[
{
'Id': 'm1',
'MetricStat': {
'Metric': {
'Namespace': namespace,
'MetricName': metric_name
},
'Period': 300, # 5 minutes
'Stat': 'Average', # Can be 'Average', 'Minimum', 'Maximum', etc.
},
'ReturnData': True,
},
],
StartTime=start_time,
EndTime=end_time
)
return response['MetricDataResults'][0]['Values']
# Example usage: Fetch memory usage metrics for the last hour
metric_name = 'MemoryUtilization'
namespace = 'CWAgent'
end_time = '2020-10-13T12:00:00Z' # Current time
start_time = '2020-10-13T11:00:00Z' # One hour ago
memory_utilization_data = get_cloudwatch_metrics(metric_name, namespace, start_time, end_time)
# Convert the data to numpy array
memory_utilization_array = np.array(memory_utilization_data).reshape(-1, 1)
# Initialize Isolation Forest model
isolation_forest = IsolationForest(contamination=0.05) # Contamination represents the proportion of outliers
# Fit the model to the memory utilization data
isolation_forest.fit(memory_utilization_array)
# Predict anomalies (outliers)
anomaly_predictions = isolation_forest.predict(memory_utilization_array)
# Print the indices of anomalies (where prediction is -1)
anomaly_indices = np.where(anomaly_predictions == -1)[0]
print("Anomaly indices:", anomaly_indices)
Hope the above code helps cloud engineers to apply the same technique in their cloud operations monitoring solutions.
By leveraging Isolation Forest in conjunction with AWS CloudWatch, cloud engineers can enhance their services monitoring capabilities and gain deeper insights into system performance. This solution should enable developers to detect anomalies effectively.