Don't neglect the data engineering,
Robson Júnior (~bsao) |
Nowadays modern data engineering requires creativity to design reliable data stores, pipelines and architecting distributed systems. Data Engineering teams tend to be the backbone of any company and Python definitely will leverage you and the company to the success.
Tools like Airflow, Luigi, PySpark, Dask and Pandas are essential tools that compose any good Python-based data architecture.
The presentation is divided into 3 concise parts: introduction to modern data engineering, python data engineering world and the career of a data engineer. We will discuss the topics:
- The Python-based data tools;
- Building reliable data processing pipelines and architectures.
- Using software engineer skills to leverage your data engineer career.
Basic python and software engineering knowledge.
I am a passionate Software Engineer focused on help companies and communities to build great software products for more than 15 years. 6 years ago I transitioned my career to be a data engineer when I started helping several companies to create a data-driven culture and use data as the backbone of their decisions. Also, I am deeply involved in software development communities in Brazil where since 2010 I organised several conferences like RuPy Brazil(www.rupy.com.br), AgileVale (www.agilevale.com.br), DevInVale, Startup Weekends and a variety of meetups.