Unit Testing for Pyspark
Unit testing for pyspark code.
- Using pytest for testing PySpark code
- Utilising mocks and pytest-spark
- Testing python UDF
- Knowledge of Python and PySpark
- Knowledge of unit testing
Anirudh, Working as a Sr. AI Engineer at a MNC.
Have experience working on data engineering and sciences.
I have spoken at BangPyper meetup earlier and also hosted Pune Stage at PyCon India last year.