• Designed and implemented an analytics platform using a Kappa system architecture with MariaDB, Python, SQL, and Jupyter Notebook, achieving a 98% uptime for 20 smart homes in Colorado and Massachusetts.

Technologies : Python, Plotly Dash, MariaDB, MySQL, Jupyter Notebook

• Established a comprehensive health, drug, biochemical, and bioinformatic Data Lake, aggregating data from diverse Relational Database Management Systems (RDBMS), Graph, and NoSQL-based databases on AWS, accumulating 100 TB of high-quality data.
• Developed Extract, Transform, and Load (ETL) pipelines on the AWS cloud, leveraging Redshift, SQS, Lambda, Batch, EMR, EC2, PySpark, Athena, Glue, and Boto3 to process a substantial 100 TB of data.
• Created multithreaded web scrapers to extract data from a variety of sources and file formats, including CSV, JSON, XML, Parquet, ORC, Avro, API, and FTP servers using Python to collect 5 GB data daily
• Orchestrated Continuous Improvement and automated test-driven development (TDD) workflows using GitHub actions, pytest, pylint, and py-coverage metrics, leading to an outstanding 100% enhancement in code quality.
• Utilized Python packages such as Pandas, NumPy, Scikit-learn, Matplotlib, Seaborn, Folium, Requests, Beautiful Soup, ElementTree, LXML, and Multiprocess for data analysis and processing.
• Implemented a robust data pipeline framework, optimizing data processing and transformation workflows.

Technologies : Python, API, AWS: S3, EMR, Athena, Glue, Redshift, Lambda, Batch, PySpark, Shell Scripting, Git, GitHub, Packages: Pandas, Requests, BeautifulSoup, Multiprocess, Pytest

• Spearheaded the construction of scalable (Extract, Transform, Load) ETL/ELT data pipelines for 5 U.S. healthcare initiatives, enabling large-scale data transformations on cloud and on-premise data warehouses.
• Successfully migrated 112 TB of data from on-premises Hadoop clusters to AWS Cloud and Snowflake, resulting in substantial cost savings.
• Pioneered an automated data quality framework in Spark Scala, reducing data errors by 35% and leading to $7000 quarterly savings.
• Executed Agile DevOps practices, ensuring code quality, version control, continuous integration, and continuous deployment (CI/CD) for 4 projects using tools such as Maven, Git, Bitbucket, Atlassian Bamboo, Jira, and Confluence.
• Proficient in stakeholder interaction, business requirements gathering, data analysis, design document creation, release management, source control management, code migration, and code reviews.

Technologies: Hadoop, Spark, Scala, Snowflake, AWS: RDS, S3, EMR, Athena, Hive, Impala, Unix, Shell scripting, Control M, Bamboo, Git, Bitbucket, Maven, Eclipse, Cloudera distribution

• Delivered an analytics product that contributed to a 12% annual increase in pharmaceutical finance sales.
• Designed and implemented a highly efficient data pipeline capable of processing 1.5 TB of data daily, streamlining data processing and transformation.
• Achieved seamless data transfer of 26 TB between Hadoop and MySQL RDBMS using Sqoop, improving data accessibility.
• Implemented best practices and performance tuning in Apache Spark jobs, resulting in a remarkable 60% reduction in response times for Spark, SQL queries, and Sqoop processes.
• Redesigned Data Lake to use Parquet, and Snappy compression to cut 30% storage and compute costs
• Performed Partitioning, Bucketing, Join optimizations, and Compression in Hive
• Applied advanced dimensional data modeling techniques, including Star schema, Kimball, and Inmon, resulting in a remarkable 20% improvement in data warehouse scalability.
• Established robust data management, governance, and data quality standards, ensuring the reliability and accuracy of datasets for data warehousing and decision-making processes.

Technologies: Hadoop, Sqoop, Hive, Impala, Shell scripting, MySQL, Spark, Scala, SQL, SonarQube, Flume, Unix, Git

Central Data Repository for MIT, Manipal:

• Delivered a web application with its main objectives to serve as a means of data entry, to collect the required data, to analyze the given data, and finally to generate reports dynamically according to the custom report format requirements of the user. The data was loaded from the databases using Sqoop and analyzed using a Hadoop cluster. The reports are generated after querying using Hive and displayed in the web application.

Project Management System:

• Developed a web application that enabled the interaction between different users of different departments and their respective projects while accessing their functions on a large scale.