Djim DIOP
Dubai
Summary
Lead data engineer with 13+ years of experience in development, analysis, testing, and support, specializing in big data technologies such as Spark, Kafka, Azure, Solr, and SQL. Successfully implemented data pipeline projects from inception to production, showcasing strong analytical skills and adaptability in complex environments. Certified in Spark and Azure data engineering, with hands-on expertise in designing, developing, and maintaining scalable data architectures. Proficient in SQL, Scala, Golang, and cloud platforms, optimizing data workflows and ensuring data integrity to meet business objectives.
Overview
14
14
years of professional experience
1
1
Certification
Work History
Lead Data Engineer
Dscale
Dubai
03.2021 - Current
- Designed and implemented real-time data platforms from scratch.
- Led the technical architecture for Kafka Streams applications.
- Built event-driven pipelines for e-commerce systems.
- Developed APIs in Golang for data access and processing.
- Designed and optimised search solutions (Solr / Algolia) for high-performance APIs..
- Built connectors (source and sink) for real-time ingestion..
- Managed data models for large-scale e-commerce systems.
- Deployed applications using Docker and Kubernetes.
- Improved system scalability and reduced latency for real-time data processing.
- Enabled faster product search and filtering (search optimisation).
- Provided mentorship and guidance to junior data engineers, fostering professional development and skill advancement.
Data Engineer
ZAND Bank
Dubai
12.2019 - 03.2021
- Development of real-time applications from scratch.
- Technical architecture for Kafka Streams applications.
- Development of Kafka Stream applications.
- Development of source connectors to ingest data from SQL Server databases to Kafka topics.
- Creating Debezium properties files.
- Development of data processing ETL with Spark.
- Creating sink connectors to ingest error data.
- Analysis and correction of anomalies.
- Migration from JSON Serde to Avro schema registry and handling connectors.
- Creating unit tests.
- Application deployment in QA and production with Docker and Kubernetes.
- Collaborated with cross-functional teams to identify and translate business requirements into technical specifications.
- Managed database projects from start to completion, with focus on using optimum practices for designing, constructing, modifying, integrating, implementing and testing software.
Data Engineer
Societe Generale
Paris
08.2016 - 11.2019
- Definition of Big Data technical solutions for different use cases
- Functional and technical specifications
- Big data model design and development of extraction, processing and data transformation
- Development of Spark Streaming programs
- Design and development of data processing modules in Spark
- Performance Tuning for Spark applications
- Migration of Spark programs from Spark 1.6 to Spark 2.2.
- Data integration in Hive tables
- Creation and management of indexes with Elasticsearch and Kibana
- Creation of dashboards in Kibana
- Creating Oozie workflows for scheduling programs.
Data Engineer/Business Intelligence
Capgemini
Rennes
04.2012 - 04.2016
- Feasibility analysis to migrate business intelligence applications to big data.
- Big Data architecture and solution definition.
- Design and development of data processing modules in Spark.
- Loading flat data to the Cassandra database.
- Creation of Spark User user-defined functions for Big Data transformations.
- Design of data warehouses and data marts.
- Loading data from flat files and Excel sheets into the Oracle database using SQL.
- Creating SSIS packages to load data to a centralised SQL Server database from diversified data sources.
- Creating and/or upgrading SQL stored procedures, tables, and queries.
- Conducted thorough data quality checks, rectifying inconsistencies and ensuring reliability of information.
- Creating testing plans and scripts based on requirements.
- Designed and implemented scalable data pipelines, integrating diverse data sources for streamlined analysis.
- Production and delivery of installation documents for pre-production.
- Production and delivery of installation documents for production.
Education
Double Degree, MSc - Business Intelligence
Grenoble Management School-ESC Grenoble
Grenoble07.2011
Masterʼs Degree - information systems for bank, finance and industry
ESIEA (École Supérieure d'Informatique, d'Électronique et Automatique)
Laval/Paris07.2010
Skills
Data processing
- Apache Spark (PySpark and Scala)
- Databricks
- Batch and stream processing
- Data transformation and optimisation
Data pipelines
- ETL/ELT pipeline design
- Data ingestion (Batch and Real-time)
- Azure Data Factory
- Workflow orchestration
Performance tuning and optimisation
- Spark performance tuning
- Query optimisation
- Data partitioning strategies
Databases
- NoSQL Databases
- Relational databases
- Columnar databases: ClickHouse
Data streaming
- Kafka CDC (Debezium, Confluent)
- Kafka streaming
Programming
- Python
- Scala
- SQL
- Go (Golang)
Cloud
- Azure
DevOps
- Docker
- Kubernetes
Hobbies and Interests
- Data
- Sports
Websites, Portfolios and Profiles
https://www.linkedin.com/in/djim-diop-80723346/
Languages
English
Proficient
C2
French
Proficient
C2
Certification
- Spark certified
- Azure Data Engineering certified
Hobbies and interests
- Data
- Exploring new technologies
- Football
Timeline
Lead Data Engineer
Dscale
03.2021 - Current
Data Engineer
ZAND Bank
12.2019 - 03.2021
Data Engineer
Societe Generale
08.2016 - 11.2019
Data Engineer/Business Intelligence
Capgemini
04.2012 - 04.2016
Double Degree, MSc - Business Intelligence
Grenoble Management School-ESC Grenoble
Masterʼs Degree - information systems for bank, finance and industry
ESIEA (École Supérieure d'Informatique, d'Électronique et Automatique)