View all jobs
Data Engineer Python Databricks
New York, NY · Information TechnologyOur client, a major bank in New York City, is looking for talented Senior Data Engineer Consultant with strong Python and Databricks.
Hybrid commute, 3 days on-site.
Consulting position, initial term is about 6 months with possible extensions and option to hire.
Hourly Rate: $90-100/h on W2. C2C is also an option.
Please read the description below and to be considered immediately email your resume to igork@brainsworkgroup.com
Senior Data Engineer - Python and Databricks
To play a crucial role in designing, implementing, and maintaining data pipelines and infrastructure for our big data projects.
Need expertise in Python, Spark cluster management, data science, big data, Snowflake, REST API development, and knowledge of Databricks and Delta Lake.
Major Responsibilities:
- Design, develop, and implement scalable data pipelines and ETL processes using Python, and Spark.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and design efficient solutions.
- Manage and optimize Spark clusters to ensure high performance and reliability.
- Perform data exploration, data cleaning, and data transformation tasks to prepare data for analysis and modeling. Snowflake is a must.
- Develop and maintain data models and schemas to support data integration and analysis.
- Implement data quality and validation checks to ensure accuracy and consistency of data.
- Utilize REST API development skills to create and integrate data services and endpoints for seamless data access and consumption.
- Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks and issues.
- Stay updated with the latest technologies and trends in big data, data engineering, data science, and REST API development, and provide recommendations for process improvements.
- Mentor and guide junior team members, providing technical leadership and sharing best practices.
Requirements:
Please email your resume or
Use this link to apply directly:
https://brainsworkgroup.catsone.com/careers/index.php?m=portal&a=details&jobOrderID=16260907
Or email: igork@brainsworkgroup.com
Check ALL our Jobs: https://brainsworkgroup.catsone.com/careers
Keywords: Java python ETL SSIS Informatica REST spark snowflake delta sql azure aws cloud bigquery warehousing spring
Hybrid commute, 3 days on-site.
Consulting position, initial term is about 6 months with possible extensions and option to hire.
Hourly Rate: $90-100/h on W2. C2C is also an option.
Please read the description below and to be considered immediately email your resume to igork@brainsworkgroup.com
Senior Data Engineer - Python and Databricks
To play a crucial role in designing, implementing, and maintaining data pipelines and infrastructure for our big data projects.
Need expertise in Python, Spark cluster management, data science, big data, Snowflake, REST API development, and knowledge of Databricks and Delta Lake.
Major Responsibilities:
- Design, develop, and implement scalable data pipelines and ETL processes using Python, and Spark.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and design efficient solutions.
- Manage and optimize Spark clusters to ensure high performance and reliability.
- Perform data exploration, data cleaning, and data transformation tasks to prepare data for analysis and modeling. Snowflake is a must.
- Develop and maintain data models and schemas to support data integration and analysis.
- Implement data quality and validation checks to ensure accuracy and consistency of data.
- Utilize REST API development skills to create and integrate data services and endpoints for seamless data access and consumption.
- Monitor and troubleshoot data pipeline performance, identifying and resolving bottlenecks and issues.
- Stay updated with the latest technologies and trends in big data, data engineering, data science, and REST API development, and provide recommendations for process improvements.
- Mentor and guide junior team members, providing technical leadership and sharing best practices.
Requirements:
3+ years of professional experience in data engineering, working with Python, Spark, and big data technologies.
- Strong programming skills in Python, with expertise in building scalable and maintainable code. Java is a plus
- Proven experience in Spark cluster management, optimization, and performance tuning.
- Solid understanding of data science concepts and experience working with data scientists and analysts.
- Proficiency in SQL and experience with relational databases (e.g., Snowflake, Delta Tables). Snowflake is a MUST.
- Experience in designing and developing REST APIs using frameworks such as Flask or Spring.
- Familiarity with cloud-based data platforms (e.g.Azure)
- Experience with data warehousing concepts and tools (e.g., Snowflake, BigQuery) is a plus.
- Strong problem-solving and analytical skills, with the ability to tackle complex data engineering challenges.
- Excellent communication and collaboration skills, with the ability to work effectively in a team-oriented environment.
Education:
- Strong programming skills in Python, with expertise in building scalable and maintainable code. Java is a plus
- Proven experience in Spark cluster management, optimization, and performance tuning.
- Solid understanding of data science concepts and experience working with data scientists and analysts.
- Proficiency in SQL and experience with relational databases (e.g., Snowflake, Delta Tables). Snowflake is a MUST.
- Experience in designing and developing REST APIs using frameworks such as Flask or Spring.
- Familiarity with cloud-based data platforms (e.g.Azure)
- Experience with data warehousing concepts and tools (e.g., Snowflake, BigQuery) is a plus.
- Strong problem-solving and analytical skills, with the ability to tackle complex data engineering challenges.
- Excellent communication and collaboration skills, with the ability to work effectively in a team-oriented environment.
Education:
- Master’s degree in Computer Science, Data Science, or a related field strongly preferred
Please email your resume or
Use this link to apply directly:
https://brainsworkgroup.catsone.com/careers/index.php?m=portal&a=details&jobOrderID=16260907
Or email: igork@brainsworkgroup.com
Check ALL our Jobs: https://brainsworkgroup.catsone.com/careers
Keywords: Java python ETL SSIS Informatica REST spark snowflake delta sql azure aws cloud bigquery warehousing spring