Data Engineer at 10pearls. Utilizing Apache Spark, MapReduce, Livy, Airflow, pySpark, s3 Bucket, Snowflake, Docker. Responsibilities Moving complex Structural data(SQL, mySQL, oracle, sftp) using pyspark remove redundancies and save to s3 bucket and snowflake. Daily jobs are monitored/scheduled using Airflow Dags. Spark jobs are submitted using Livy. ©