{"id":5559,"date":"2026-02-02T14:56:45","date_gmt":"2026-02-02T14:56:45","guid":{"rendered":"https:\/\/cephasconsult.biz\/?post_type=job_listing&p=5559"},"modified":"2026-03-05T00:30:14","modified_gmt":"2026-03-05T00:30:14","slug":"lead-ii-software-engineering-aws-apache-spark-pyspark-scala-apache-kafka-47292","status":"expired","type":"job_listing","link":"https:\/\/cephasconsult.biz\/?post_type=job_listing&p=5559","title":{"rendered":"Lead II – Software Engineering – AWS, Apache Spark (PySpark\/Scala), Apache Kafka 47292"},"content":{"rendered":"

Positions:3 <\/span><\/span>Full Time<\/span><\/p>\n

\n
Experience<\/div>\n
5 – 8 Years<\/div>\n<\/div>\n
<\/div>\n
\n
\n
\n
\n
\n
\n
\n

Job Summary<\/h3>\n

We are seeking a skilled Data Engineer to design, build, and optimize scalable data pipelines and cloud-based data platforms. The role involves working with large-scale batch and real-time data processing systems, collaborating with cross-functional teams, and ensuring data reliability, security, and performance across the data lifecycle.<\/p>\n

Key Responsibilities<\/p>\n

ETL Pipeline Development & Optimization<\/h3>\n
    \n
  • Design, develop, and maintain complex end-to-end ETL pipelines for large-scale data ingestion and processing.<\/li>\n
  • Optimize data pipelines for performance, scalability, fault tolerance, and reliability.<\/li>\n<\/ul>\n

    Big Data Processing<\/h3>\n
      \n
    • Develop and optimize batch and real-time data processing solutions using Apache Spark (PySpark\/Scala)<\/strong> and Apache Kafka<\/strong>.<\/li>\n
    • Ensure fault-tolerant, scalable, and high-performance data processing systems.<\/li>\n<\/ul>\n

      Cloud Infrastructure Development<\/h3>\n
        \n
      • Build and manage scalable, cloud-native data infrastructure on AWS<\/strong>.<\/li>\n
      • Design resilient and cost-efficient data pipelines adaptable to varying data volume and formats.<\/li>\n<\/ul>\n

        Real-Time & Batch Data Integration<\/h3>\n
          \n
        • Enable seamless ingestion and processing of real-time streaming and batch data sources (e.g., AWS MSK<\/strong>).<\/li>\n
        • Ensure consistency, data quality, and a unified view across multiple data sources and formats.<\/li>\n<\/ul>\n

          Data Analysis & Insights<\/h3>\n
            \n
          • Partner with business teams and data scientists to understand data requirements.<\/li>\n
          • Perform in-depth data analysis to identify trends, patterns, and anomalies.<\/li>\n
          • Deliver high-quality datasets and present actionable insights to stakeholders.<\/li>\n<\/ul>\n

            CI\/CD & Automation<\/h3>\n
              \n
            • Implement and maintain CI\/CD pipelines using Jenkins<\/strong> or similar tools.<\/li>\n
            • Automate testing, deployment, and monitoring to ensure smooth production releases.<\/li>\n<\/ul>\n

              Data Security & Compliance<\/h3>\n
                \n
              • Collaborate with security teams to ensure compliance with organizational and regulatory standards (e.g., GDPR, HIPAA).<\/li>\n
              • Implement data governance practices ensuring data integrity, security, and traceability.<\/li>\n<\/ul>\n

                Troubleshooting & Performance Tuning<\/h3>\n
                  \n
                • Identify and resolve performance bottlenecks in data pipelines.<\/li>\n
                • Apply best practices for monitoring, tuning, and optimizing data ingestion and storage.<\/li>\n<\/ul>\n

                  Collaboration & Cross-Functional Work<\/h3>\n
                    \n
                  • Work closely with engineers, data scientists, product managers, and business stakeholders.<\/li>\n
                  • Participate in agile ceremonies, sprint planning, and architectural discussions.<\/li>\n<\/ul>\n

                    Skills & Qualifications<\/p>\n

                    Mandatory (Must-Have) Skills<\/h3>\n
                      \n
                    1. AWS Expertise<\/strong>\n
                        \n
                      • Hands-on experience with AWS Big Data services such as EMR, Managed Apache Airflow, Glue, S3, DMS, MSK, and EC2<\/strong>.<\/li>\n
                      • Strong understanding of cloud-native data architectures.<\/li>\n<\/ul>\n<\/li>\n
                      • Big Data Technologies<\/strong>\n
                          \n
                        • Proficiency in PySpark or Scala Spark<\/strong> and SQL<\/strong> for large-scale data transformation and analysis.<\/li>\n
                        • Experience with Apache Spark<\/strong> and Apache Kafka<\/strong> in production environments.<\/li>\n<\/ul>\n<\/li>\n
                        • Data Frameworks<\/strong>\n
                            \n
                          • Strong knowledge of Spark DataFrames and Datasets<\/strong>.<\/li>\n<\/ul>\n<\/li>\n
                          • ETL Pipeline Development<\/strong>\n
                              \n
                            • Proven experience in building scalable and reliable ETL pipelines for both batch and real-time<\/strong> data processing.<\/li>\n<\/ul>\n<\/li>\n
                            • Database Modeling & Data Warehousing<\/strong>\n
                                \n
                              • Expertise in designing scalable data models for OLAP and OLTP<\/strong> systems.<\/li>\n<\/ul>\n<\/li>\n
                              • Data Analysis & Insights<\/strong>\n
                                  \n
                                • Ability to perform complex data analysis and extract actionable business insights.<\/li>\n
                                • Strong analytical and problem-solving skills with a data-driven mindset.<\/li>\n<\/ul>\n<\/li>\n
                                • CI\/CD & Automation<\/strong>\n
                                    \n
                                  • Basic to intermediate experience with CI\/CD pipelines<\/strong> using Jenkins<\/strong> or similar tools.<\/li>\n
                                  • Familiarity with automated testing and deployment workflows.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                                    Good-to-Have (Preferred) Skills<\/h3>\n
                                      \n
                                    • Knowledge of Java<\/strong> for data processing applications.<\/li>\n
                                    • Experience with NoSQL databases<\/strong> (e.g., DynamoDB, Cassandra, MongoDB).<\/li>\n
                                    • Familiarity with data governance frameworks<\/strong> and compliance tooling.<\/li>\n
                                    • Experience with monitoring and observability tools such as AWS CloudWatch, Splunk, or Dynatrace<\/strong>.<\/li>\n
                                    • Exposure to cost optimization strategies for large-scale cloud data platforms.<\/li>\n<\/ul>\n

                                      Skills:<\/strong>big data,scala spark,apache spark,etl pipeline development,<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"template":"","meta":{"_job_location":"Hyderabad, India","_application":"hrm@cephasconsult.biz","_company_name":"","_company_website":"","_company_tagline":"","_company_twitter":"","_company_video":"","_filled":0,"_featured":0,"_remote_position":0,"_job_salary":"","_job_salary_currency":"","_job_salary_unit":""},"job-types":[],"class_list":["post-5559","job_listing","type-job_listing","status-expired","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/cephasconsult.biz\/index.php\/wp-json\/wp\/v2\/job-listings\/5559","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cephasconsult.biz\/index.php\/wp-json\/wp\/v2\/job-listings"}],"about":[{"href":"https:\/\/cephasconsult.biz\/index.php\/wp-json\/wp\/v2\/types\/job_listing"}],"author":[{"embeddable":true,"href":"https:\/\/cephasconsult.biz\/index.php\/wp-json\/wp\/v2\/users\/1"}],"wp:attachment":[{"href":"https:\/\/cephasconsult.biz\/index.php\/wp-json\/wp\/v2\/media?parent=5559"}],"wp:term":[{"taxonomy":"job_listing_type","embeddable":true,"href":"https:\/\/cephasconsult.biz\/index.php\/wp-json\/wp\/v2\/job-types?post=5559"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}