You work for a web retailer where you need to analyze data produced for your company by an outside market data provider.
You need to produce recommendations based on patterns in user preferences by demographic found in the supplied data.
You have stored the data in one of your company's S3 buckets.
You have created a Glue crawler that you have configured to crawl the data on S3 and you have written a custom classifier.
Unfortunately, the crawler failed to create a schema.
Why might the Glue crawler have failed in this way?
Click on the arrows to vote for the correct answer
A. B. C. D.Answer: C.
Option A is incorrect.
This configuration option is used to exclude objects from the crawler.
From the help text on the Add a Data Store screen in the Add Crawler console flow: “The exclude pattern is relative to the include path.
Objects that match the exclude pattern are not crawled.
For example, with include path s3://mybucket/ and exclude pattern,mydir/**
Then all objects in the include path below the mydir directory are skipped.
In this example, any object whose path matchess3://mybucket/mydir/** is not crawled.
For more information about patterns, see Cataloging Tables with a Crawler”
Option B is incorrect.
The IAM role assigned to your crawler needs exactly this managed policy and S3 bucket access.
From the Choose an IAM Role screen on the Add Crawler console flow: “Create an IAM role named ‘AWSGlueServiceRole-rolename' and attach the AWS managed policy, AWSGlueServiceRole, plus an inline policy that allows read access to: s3://yourbucketname”
Option C is correct.
The data from the market data provider did not match with certainty any of the built-in classifiers that are part of Glue or your custom classifier.
Therefore, Glue returned the default classification string of UNKNOWN.
(See the Amazon Glue doc Adding Classifiers to a Crawler)
Option D is incorrect.
This setting allows you to group compatible schemas.
Choosing this option would not prevent the crawler from producing the schema.
From the Configure the Crawler's Output screen in the Add Crawler console flow: “This crawler configuration groups compatible schemas into a single table definition across all S3 objects under the provided include path.
Other criteria will still be considered to determine proper grouping.”
Reference:
Please see the AWS developer guides AWS Glue: How It Works and AWS Glue Concepts.
The Glue crawler failed to create a schema for the data stored in the S3 bucket. There can be several reasons for this failure, but we will analyze each answer option to understand the most likely reason.
A. You did not add an exclude pattern when you configured the data store.
Exclude patterns are used to exclude files or folders from being crawled by the Glue crawler. If the exclude pattern is not set correctly, the crawler may try to crawl files that are not in the correct format, which can lead to a failure in schema creation. However, this is not the most likely reason for the failure in this scenario as the question does not provide any indication of incorrect file formats.
B. The IAM role you assigned to the crawler has the AWSGlueServiceRole managed policy attached plus an inline policy that allows read access to your S3 bucket.
AWS Glue crawlers require an IAM role to access resources such as S3 buckets. The AWSGlueServiceRole managed policy provides the necessary permissions to perform Glue operations, but it may not provide sufficient permissions to access the S3 bucket. Therefore, an inline policy is usually created to grant read access to the specific S3 bucket where the data is stored. However, this answer option does not provide any indication of the specific permission issues that may have caused the failure.
C. All the classifiers returned a certainty of 0.0.
Classifiers are used to categorize data during the crawling process, and they return a certainty score that indicates how confident the classifier is in its results. If all classifiers return a certainty of 0.0, it means that they were unable to categorize the data, and the Glue crawler would not be able to create a schema. However, this scenario is unlikely as it would require all classifiers to fail to categorize the data, which is highly unlikely.
D. You chose to create a single schema for each S3 path.
Glue crawlers can create one or more schemas depending on the data stored in the S3 bucket. If a single schema is chosen for each S3 path, the crawler may not be able to account for variations in the data, leading to schema creation failure. Therefore, this option is the most likely reason for the Glue crawler's failure in this scenario.
In summary, option D is the most likely reason for the Glue crawler's failure in creating a schema for the data stored in the S3 bucket.