Exam Certified Associate Developer for Apache Spark topic 1 question 21 discussion

Answer: E I tried it with the code below, all other options raised an error: # register UDF with udf function from pyspark.sql import SparkSession from pyspark.sql.functions import col, udf from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, 3, 20000, "A"), (1, 1, 50000, "A"), (2, 2, 70000, "A"), (3, 5, 10000, "B"), (4, 4, 100000, "B"), ] storesDF = spark.createDataFrame(data, ["storeID", "customerSatisfaction", "sqft", "division"]) try: storesDF.filter(col("sqft") <= 25000 | col("customerSatisfaction") >= 30).show() except Exception as e: print(e) try: storesDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30)).show() except Exception as e: print(e)

Option E, storesDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30)), is the correct option. It uses the filter() operation with the conditions (col("sqft") <= 25000) | (col("customerSatisfaction") >= 30) to filter the rows where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30.

E has the right syntax, logic, operator and correct number of parentheses. All of the others falter in one of these respects.

Should be A. Tested it in communitity edition with 2 filters.

The correct code block to return a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30 is: storesDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30)) Option A uses a single pipe (|) instead of the correct syntax of two vertical bars (||) to represent "OR" logic, and also uses the wrong syntax for column referencing. Option B uses the correct or operator, but also uses the wrong syntax for column referencing. Option C uses the correct operator and syntax for column referencing, but does not use the col() function to reference column names. Option D uses the col() function, but also uses the wrong syntax for column referencing. Option E uses the correct syntax for both column referencing and logical operator, and correctly specifies the parentheses to ensure the proper order of operations. Therefore, the correct answer is E. storesDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30))