Published inDev Genius·Aug 7Snowflake Performance Tuning: Efficiently Query External Stage(AWS S3) Data using Partitioned External Tables.In this Cloud Era, most of the company is storing the raw data in the Data Lake captured from the multiple sources. Then this data can be ingested into cloud based databases like Snowflake by transforming the raw Data into useful information for business analysis purpose by end users. It…Snowflake14 min readSnowflake14 min read
Jul 26Data Pipeline — Replicate data from MySQL to Snowflake with AWS Kinesis, Binlog Streamer Python Library & SnowpipeIn this post we will see how to replicate the data automatically from MySQL (OLTP) database to Snowflake (OLAP) Cloud Data-Warehouse whenever there are some data changes happened in the source database (OLTP). This article is little lengthy because I tried to explain each and every steps in detail. So…AWS18 min readAWS18 min read
Jul 26Snowflake — Replicate CDC log files (JSON Format) from Data Lake (AWS S3) to Snowflake Tables.Now a days many companies are moving their Data Warehousing & Business Intelligence(BI) solution into the cloud computing-based technologies. Snowflake Data Cloud is a go to option for this because of it’s features & advantages like: Zero infrastructure & management overhead. Consumption-based pricing. Unlimited compute and storage scalability Native SQL…Snowflake12 min readSnowflake12 min read
Jul 9Read File Data from S3 using Python AWS LambdaIn this post we will see how to automatically trigger the AWS Lambda function which will read the files uploaded into S3 bucket and display the data using the Python Pandas Library. We will use boto3 API’s to read the files from S3 Bucket. Steps followed to achieve this: 1…AWS7 min readAWS7 min read
Jul 9Snowflake & AWS Secure Integration Framework via Integration ObjectsSnowflake, as we all know is a data warehouse built for the cloud. It allows the users to read/write the data from/to cloud storage. So it is essential to have secure integration between the platforms. When it comes to security (In AWS), we should always Apply least-privilege permissions & follow…Data Engineering9 min readData Engineering9 min read
