Senior Data Engineer-REF7014
About Creative Software
Founded in 1999, Creative Software is a global technology enabler and pioneer in Sri Lanka’s tech industry. We manage teams of high-performing, dedicated software engineers for our global clientele, providing them with end-to-end software development and maintenance services through team augmentation.
Embark on a Creative career that offers a dynamic work environment, competitive intern allowance, and room for personal and professional growth. Be part of a community of professionals, contributing to an inclusive culture that provides you with valuable local and global exposure. At Creative, we offer a variety of spaces that support work-life balance and integrates wellness into our workspace experience.
About the Role:
We are seeking a talented and proactive Senior Data Engineer to design, build, and maintain the data infrastructure that powers our business insights and advanced analytics. As a key member of our technical team, you will be responsible for developing robust data pipelines, implementing efficient ETL processes, and optimizing database systems to deliver actionable insights. You will collaborate closely with cross-functional teams, including data scientists, analysts, and business stakeholders, ensuring that data solutions align with organizational objectives.
Key Responsibilities:
1. Data Architecture and Pipeline Development
o Design and implement scalable, high-performance data architectures to meet analytical and operational needs.
o Build, test, and deploy reliable data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
o Continuously optimize workflows to enhance processing efficiency and minimize latency.
2. Database Management and Optimization
o Manage and maintain databases, ensuring data integrity, security, and scalability.
o Create, modify, and troubleshoot SQL queries and stored procedures for optimal performance.
o Assess and implement appropriate database technologies (e.g., SQL, NoSQL) based on project requirements.
3. ETL Processes and Data Transformation
o Develop and maintain robust ETL (Extract, Transform, Load) workflows to ensure seamless data integration across systems.
o Automate ETL processes to improve reliability and efficiency.
4. Big Data and Advanced Processing
o Leverage big data technologies (e.g., Apache Hadoop, Spark) to analyze and process large-scale datasets.
o Design distributed computing solutions to address complex data challenges.
5. Data Quality, Governance, and Security
o Enforce data quality standards to ensure accuracy and consistency across datasets.
o Implement governance policies to comply with security, privacy, and regulatory standards.
o Monitor and enhance system performance, resolving issues as they arise.
6. Collaboration and Stakeholder Engagement
o Partner with data scientists, analysts, and business teams to gather data requirements and deliver tailored solutions.
o Translate complex technical concepts into accessible terms for non-technical stakeholders.
o Contribute to documentation and knowledge-sharing initiatives.
7. Cloud and Infrastructure Management
o Deploy and manage cloud-based data storage, processing, and analytics infrastructure using platforms like Azure, AWS, or Google Cloud.
o Stay updated with emerging cloud technologies and implement best practices.
Key Qualifications:
· Bachelor’s or master’s degree in computer science, Information Technology, or a related field.
· 3-5 years of experience as a Data Engineer or in a similar role.
· Proficiency in programming languages such as Python, Java, or Scala.
· Strong expertise with database technologies (SQL, NoSQL, TSQL) and data warehousing solutions.
· Hands-on experience with tools like Azure Spark, Databricks, Azure Synapse Analytics, Azure Data Factory, and SQL Enterprise Suite (SSIS, Power BI, SSAS).
· Knowledge and experience in data warehouse modeling approaches, including Star Schema and the Kimball Approach.
· Experience with big data tools like Apache Spark, Kafka, or Hadoop.
· Familiarity with cloud platforms (Azure, AWS, Google Cloud) and associated data services.
· Strong analytical and problem-solving skills with the ability to manage multiple priorities.
· Knowledge of data security, governance, and compliance best practices.
· Excellent communication and interpersonal skills to effectively collaborate with diverse teams.
Preferred:
· Experience with data visualization tools (e.g., Tableau, Power BI).
· Familiarity with machine learning workflows and tools (e.g., TensorFlow, PyTorch).
· Understanding of DevOps practices and CI/CD pipelines for data systems.