Your current log analysis application takes more than four hours to generate a report of the top 10 users of your web application.
You have been asked to implement a system that can report this information in real-time, ensure that the report is always up to date, and handle increases in the number of requests to your web application.
Choose the option that is cost-effective and can fulfill the requirements.
Click on the arrows to vote for the correct answer
A. B. C. D.Answer - C.
When you see Amazon Kinesis as an option, this becomes the ideal option to process data in real-time.
Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information.
Amazon Kinesis offers key capabilities to process streaming data cost-effectively at any scale, along with the flexibility to choose the tools that best suit the requirements of your application.
With Amazon Kinesis, you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, and more into your databases, data lakes, and data warehouses, or build your own real-time applications using this data.
For more information on AWS Kinesis, please refer to the below link-
https://aws.amazon.com/kinesis/Option A is incorrect because logs in CloudWatch Logs are not in real-time.
Option B is incorrect because you have to pay for S3 buckets and EC2 instances cost, which is not a cost-effective solution.
And the logs may not be in real-time.
Option D is incorrect because it is not cost-efficient to store the logs in RDS MySQL.
Option A: Publishing log data to CloudWatch Logs can provide real-time log data analysis. Auto Scaling can be used to manage load on demand, which can improve the application's scalability. However, using CloudWatch Logs to generate the report may not be the most cost-effective option. This is because CloudWatch Logs can be expensive when the number of log events increases, which can be a concern if the web application has high traffic.
Option B: Publishing log data to an Amazon S3 bucket can provide an easy and cost-effective way to store and retrieve log data. CloudFormation can be used to automate the creation of an Auto Scaling group, which can scale the post-processing application up or down based on demand. This option can be more cost-effective than using CloudWatch Logs because storing data in an S3 bucket is cheaper than using CloudWatch Logs.
Option C: Posting log data to an Amazon Kinesis data stream can provide a scalable and reliable way to process real-time streaming data. Kinesis allows for processing of large amounts of data in real-time, and can be used to subscribe the log-processing application, which can be configured to process the logging data. This option can be more suitable for high traffic web applications that generate large amounts of log data.
Option D: Creating a multi-AZ Amazon RDS MySQL cluster can provide a reliable and scalable way to store log data. Running a map reduce job to retrieve the required information on user counts can provide near real-time reports. However, this option may not be as cost-effective as other options, especially if the application generates a large amount of log data.
Based on the requirements provided, Option C (using Amazon Kinesis data stream) appears to be the most suitable option for handling real-time log analysis of a high-traffic web application. However, the choice ultimately depends on the specific requirements of the application, including factors such as cost, scalability, and reliability.