Exam Certified Data Engineer Associate topic 1 question 23 discussion

C. The table was external When dropping an external table in Spark SQL, only the metadata is removed. The actual data files remain in their original location because they are not managed by Spark but by the external source.

C is correct

THE QUESTION SHOULD BE "Which of the following describes why the metadata files still exist and the data files were deleted?"

C. The table was external The reason why the data files still exist while the metadata files were deleted is because the table was external. When a table is external in Spark SQL (or in other database systems), it means that the table metadata (such as schema information and table structure) is managed externally, and Spark SQL assumes that the data is managed and maintained outside of the system. Therefore, when you execute a DROP TABLE statement for an external table, it removes only the table metadata from the catalog, leaving the data files intact. On the other hand, for managed tables (option E), Spark SQL manages both the metadata and the data files. When you drop a managed table, it deletes both the metadata and the associated data files, resulting in a complete removal of the table.

Option C

C is the correct answer. For external tables, you need to go to the specific location using DESCRIBE EXTERNAL TABLE command and delete all files.