Score:0

Is using Azure Data Factory for transformation of Azure hosted Snowflake efficient?

us flag

I'm trying to understand the efficiency of using Azure Data Factory for transforming data within Snowflake (Azure based). We have two possible scenarios and want to pick the most efficient:

Scenario 1:

  • Data Factory orchestrates the ingestion of raw data to Azure SQL
  • Data Factory orchestrates the transformation and loading of raw data in Azure SQL to summary tables in Snowflake. Historic raw data is kept in Azure SQL.

Scenario 2:

  • Data Factory orchestrates the ingestion of raw data into Snowflake
  • Data Factory orchestrates the transformation of raw data in Snowflake to summary tables in Snowflake. Historic raw data is kept in Snowflake.

Does scenario 2 incur additional costs in egress of raw data to ADF (data sets) from Snowflake in the transformation step or does it all happen in Snowflake without data set egress?

Reading the ADF documentation, it seems the compute itself happens on the linked service (i.e. Snowflake), not within ADF itself, but does that mean that data does not leave Snowflake when ADF transforms it?

Let me know if the question isn't clear. Thanks!

Score:0
in flag
mwa

for scenario 2, data is passed to Snowflake and transformed there. ADF has only an orchestrator role here and no egress traffic with action 2.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.