Exam Certified Data Engineer Professional topic 1 question 226 discussion

B - to test functions we need to import them in our unit tests. This means that storing functions in notebooks is not a good idea. You store them separately - as "files" - and import in notebooks the same way you import them in unit tests.

Databricks Repos is the recommended way to organize and manage code, including functions and unit tests, in a scalable and maintainable way. By defining your functions and unit tests in Files in Repos, you can: Modularize Your Code: Functions can be organized into separate Python files or modules, making them reusable and easier to test. Use Standard Testing Frameworks: Frameworks like pytest or unittest can be used to write and execute unit tests against these functions. Version Control Integration: Files in Repos can be version-controlled using Git, ensuring traceability and collaboration. Production Data Testing: With proper safeguards, you can design unit tests to test production-like data while maintaining modularity and separation from production pipelines.

It is true that or Python and R notebooks, Databricks recommends storing functions and their unit tests outside of notebooks so given the question is in Python environment some could argue B being the correct solution. Nevertheless that is only an "advice" and Databricks more generally states that in general, it is a best practice to not run unit tests against functions that work with data in production. This is especially important for functions that add, remove, or otherwise change data. To protect your production data from being compromised by your unit tests in unexpected ways, you should run unit tests against non-production data. So I would got for option C. Alternatively I would have gone for B. https://docs.databricks.com/en/notebooks/testing.html?utm_source=chatgpt.com#write-unit-tests