Much has been said in the data world about ideal states: “Data at your fingertips”, “Plug-and-play insights”, “Empower your whole organization with data”, and “Democratizing insights from data”. This list is long, and to be honest, does sound great. There is value in setting such big-picture goals in an organization, but often the route to go from here to there is a lot harder than initially considered. Data Warehouses such as Snowflake are often proposed as the solution that ushers in this step-change in organizational performance. If only your data was one well-defined, easy-to-understand, definitely maintained query away, everyone would be able to incorporate it into their day-to-day decisions. At Eagle Alpha, we think Data Warehouses can be useful but often find ourselves in a chicken-and-egg situation with them. As mentioned, Data Warehouses do their best when the data is organized, but who is supposed to do that organizing and how? How do you know what to set up so that data users in your organization can hit the ground running when all you have are 300,000 poorly named CSV files of unknown content scattered across 1500 sub-directories of sub-directories? The ideal starts to drift pretty far away when that is the reality – a reality that many organizations face even with the internal data that they nominally can control the production of. Add in the requirement to incorporate external third-party sources of data, and soon addressing the ambiguity and confusion that stems from this data circus becomes the all-consuming job of many data engineering teams.
