Professional Summary
Having 8+ years of IT experience focusing on Data warehousing, Data modeling, Data integration, Data Migration using ETL Jobs, and Database administration. Using AWS SDK for python to access AWS services programmable within QLIK. Installing the Boto3 library and configuring credentials within the QLIK Firsthand experience in using Hadoop ecosystem components like Hadoop, Map Reduce, Yarn, Hive, Pig, Sqoop, HBase, Cassandra, Spark, Oozie, Zookeeper, Kafka, and Flume. Familiar with Spark streaming to receive real-time data from Kafka and store the stream data to HDFS using Scala. Experience with databases like DB2, MySQL, SQL, and MongoDB. Proficient in converting Hive/SQL queries into Spark transformations using Spark Data frames and Scala. Good understanding and knowledge of NoSQL databases like MongoDB, HBase, and Cassandra. Experienced in in workflow scheduling and locking tools/services like Oozie and Zookeeper Knowledge of ETL methods in enterprise-wide solutions, data warehousing, reporting, and data analysis. Experienced in in working with AWS using its services such as EC2, Amazon S3, ECS, Amazon RDS, VPC, IAM, Lambda, Amazon Elastic Load Balancing, Cloud Front, Auto Scaling, Cloud Watch, and Redshift. Sound knowledge in using Pig scripts for transformations, event joins filters, and pre-aggregations for HDFS storage. Hands-on experience in importing and exporting data with Sqoop to and from HDFS to RDBMS including Oracle, MySQL, and MS SQL Server Good Knowledge of UNIX Shell Scripting for automating deployments and other routine tasks. Experienced in using IDEs like Eclipse, IntelliJ, PyCharm, Jupiter, and Data bricks notebooks. Skilled in using JIRA and Rally for bug tracking and GitHub and SVN for various code reviews and unit testing Experienced in working in all phases of SDLC - both agile and waterfall methodologies. Good understanding of Agile Scrum methodology, Test Driven Development, and CI-CD. Excellent verbal and written communication skills, Ability to meet deadlines in a fast pace environment, and A strong passion for learning new things and solving problems.
Experience
Position
- General Dynamics Experience, Data Engineer Trenton, NJ 12/2020 Current
- Worked on infrastructure deployment on AWS using EC2 (Virtual Cloud Servers), RDS (Configured Relational Database Service), VPC (Virtual Private Cloud), managed the Network and its Security, Route 53, Cloud Formation, Direct Connect, AWS S3, AWS Ropeworks (operations automation), IAM, Glacier (Cloud storage) & Amazon CloudWatch Monitoring Management
- Applied AWS's CLI to automate backups of short-lived data-stores to S3 buckets, EBS and created nightly backup AMIs for the mission critical production servers
- Developed and optimized ETL workflows in both legacy and distributed environments Worked with Cloud Watch, EC2, managing securities and Elastic Load Balancing on AWS
- Practical knowledge on AWS Ops Works, CloudFormation and AWS Elastic Beanstalk
- Created SQL Server (T-SQL) stored procedures, views, and Power BI reports for project planning and regulatory reporting for storing and receiving high amounts of Data Amazon s3 simple storage service
- Developing web applications using Django framework: experience in building web applications using the Django framework
- This includes knowledge of the Model-View-Controller (MVC) architecture
- Managed mission-critical Hadoop cluster and Kafka at the production scale, especially Cloudera distribution
- Used AWS data pipeline for Data Extraction, Transformation and Loading from homogeneous or heterogeneous data sources and built various graphs for business decision-making using Python Matplotlibrary
- Mobile app development: using Django to build the backend of mobile apps
- Developing Django's REST framework to create APIs that connect their mobile apps to their databases and other web services
- Added support for Amazon AWS S3 and RDS to host static/media files and the database into Amazon Cloud
- Created Model dependency mapping by recursive tree traversal on regex extracted model ids from model descriptions
- Used HTML/CSS, JavaScript and AJAX for development of the website's user interface Developed views and templates with Django's view controller
Designed REST
- REST APIs and/or packages that abstract feature extraction and complex prediction/ forecasting algorithms on time series data
- Created and maintained data dictionaries, database schema, SQL & Python coding style/standard guides and runbooks for ETL & exposure reporting processes
- Performed data ingestion and data manipulation for Vertica, SharePoint, delimited files, APIs etc Using Python
- Performed data analysis, data cleaning, data visualization and data transformation using the SciPy stack (pandas, matplotlib, NumPy) and Power BI to discover insights in data
- Environment: Python, Django Web Framework, Pandas, Matplotlib, SciPy, NumPy, Power BI, REST API, Vertica, SharePoint, AWS (RDS, IAM, S3 Cloud Watch, Route 53, VPC, Autoscaling, Shell Scripts, RDS, SES, SQS and SNS), PyUnit, MySQL, Git, Jira, HTML, HTML5/CSS, jQuery, JavaScript, Unix/Linux, Mac, Windows environment, T-SQL, SQL Server
Data Engineer/Model Developer With PythonTroy, MI 07/2018 - 11/2020
General Dynamics
- Developed Restful Micro Services using Django and deployed on AWS servers using EBS and EC2 Worked on Flask and Snowflake queries
- Designed and maintained databases using Python and developed Python-based API (RESTful Web Service) using Flask, SQL Alchemy, and PostgreSQL
- Used Jenkins for continuous integration and delivery platform over GIT
- Automated most of the daily task using python scripting
- Involved in the CI/CD pipeline management for managing the weekly releases
- Worked on Jira for managing the tasks and improving the individual performance
- Making recommendations to the team in terms of appropriate testing techniques, shared testing tasks Lasks
- Defined different Django API profiling techniques for faster rendering information
- Later Migrated applications from Django to Flask and NoSQL (DynamoDB) to SQL(Snowflake) Used Jenkins, AWS, Bitbucket environments
- Worked on Automating, Configuring and deploying instances on AWS, Azure environments and Data centers, also familiar with EC2, Cloud watch, Cloud Formation and managing security groups on AWS
- Troubleshooted Production issues pertaining to AWS Cloud Resources and Application Infrastructure point of view
- Developed Shell scripts for cleansing and validating data files using utilities like AWK, sed, grep and other UNIX commands
- Developed Shell scripts implementing PL/SQL queries for data migration & batch processing
- Wrote python scripts to parse JSON documents and load the data in PostgreSQL database
- Used Apache Airflow programmatically to schedule and monitoring workflows and to orchestrate complex data pipelines
- Experience with building data pipelines in python/Pyspark and building python DAG in Apache Airflow
Moved
Worked with JSON based REST Web services
- I datawarehouse from on-prem teradata/postgresql to AWS Cloud
- Designed, implemented, and maintained solutions for using Docker, Jenkins, Git, and Puppet for microservices and continuous deployment
- Working on improvement of existing machine learning algorithms to extract data points accurately
- Responsible for setting up Python REST API framework using Flask
Engaged in Design, Development, Deployment, Testing and Implementation
- Developed framework for converting existing PowerCenter mappings and to PySpark (Python and Jobs
Spark
- Created Pyspark frame to bring data from DB2 to Amazon $3
- Provided guidance to development team working on PySpark as ETL platform
- Optimized the Pyspark jobs to run on Kubernetes Cluster for faster data processing
- Environment: Python 3.6, Django Framework 1.3, Flask Framework, AngularJS, CSS, DynamoDB
- MySQL, HTML5/CSS, Snowflake, Amazon Web Service (AWS), S3, EC2, PyCharm, Microsoft Visual Code, Linux, Shell Scripting.
Python DeveloperCity, STATE 05/2017 - 06/2018
Berkadia Solutions
- Participated in the complete SDLC process and used Python to develop website functionality Developed entire frontend and backend modules using Python scripting on Django Web Framework Designed and developed data management system using MySQL
- Built application logic using Python 2.7
Used Django APIs for database access
- Participated in requirement gathering and worked closely with the architect in designing and modelling
- Worked with the team to customize word press security installations
- Used Python scripts to Work with Facebook Platform Python SDK to access and post on Facebook page's wall
- Worked on the development of SQL and stored procedures on MYSQL
- Self-motivated team player with good Analytical, Logical and Problem-solving ability
- Designed and developed horizontally scalable APIs using Flask
Designed Cassandra schema for the APIs
- Designed RESTful XML web service for handling AJAX requests
- Developed remote integration with third party platforms by using RESTful web services
- Updated and maintained Jenkins for automatic building jobs and deployment
Implemented database access using Django ORM
- Backend data access modules using PL/SQL stored procedures and Oracle was designed and created
- Created RESTful API calls with the server, parse output report of excel files
- Extensively used python modules such as requests, urllib, urllib2 for web crawling
- Used Pandas API to put the data as time series and tabular format for east timestamp data manipulation and retrieval
- Used Test Driven Approach for developing the application and implemented the unit tests using Python Unit Test framework
- Used many regular expressions in order to match the pattern with the existing one and store them in database on a chronological basis
- Implemented code to perform CRUD operations on MYSQL using Toad
- Worked on deployment of the web application using the Linux server
- Environment: Python 2.7, Django Framework 1.3, CSS, PyCharm, SQL, MySQL, LAMP, jQuery, Adobe Dreamweaver, Apache Web Server
Python DeveloperSTATE 06/2015 - 04/2017
Rational Technologies
- Creating a web-based application using Python on the Django framework for data processing Implementing the preprocessing procedures along with deployment using the AWS services and creating a virtual machine using EC2
- Worked on Exploratory data analysis and performed data wrangling and data visualization Validating the data to check for the proper conversion and identifying and cleaning unwanted data, data profiling for accuracy, completeness, and consistency
- Preparing standard reports, charts, graphs, and tables from a structured data source by querying data repositories using Python and SQL
- Developed and produced a dashboard, and key performance indicators and monitor organization performance
- Define data needs, evaluate data quality, and extract/transform data for analytic projects and research
- Used Django framework for application development
- Designed and maintained databases using Python and developed Python-based API (RESTful Web Service) using Flask, SQL Alchemy, and PostgreSQL
Worked
- on server-side applications using Python programming
- Performed efficient delivery of code and continuous integration to keep in line with Agile principles Experience in Agile Methodologies, Scrum stories, and sprints experience in a Python-based environment
- Importing and exporting data between different data sources using SQL Server Management Studio Maintaining program libraries, user manuals, and technical documentation
- Environment: Python, Django, AWS, EC2, SQL, RESTful, Flask, PostgreSQL, Agile.