December 12th, 2017
Building a big data technology stack is a complex undertaking, requiring the integration of numerous different technologies for data storage, ingestion, processing, operations, governance, security and data analytics – as well as specialized expertise to make it all work. When selecting your tech stack, it is important to choose technologies that are scalable, extensible, modular and interoperable so that you have the option to incorporate new and emerging tools and technologies as they evolve. The big data landscape continues to change rapidly – so this really is critical to keep in mind to ensure you make the most of your investment.
A modern data lake infrastructure should integrate both on-premise and cloud storage. Even in today’s world, where cloud adoption seems to be the go-to strategy of every IT expert, on-prem storage and processing in reality are important to enterprise-wide data lakes, as they provide tighter control of data security and data privacy.
However, the cloud also is vital to the data lake. It offers the highly scalable and elastic storage and computing resources enterprises need for large-scale processing and data storage – without the overhead of provisioning and maintaining expensive infrastructure. Also, as big data tools and technologies continue to rapidly change, cloud-based data lakes can be used as development or test environments to evaluate new tools and technologies before bringing them to production, either in the cloud or on-prem.
The technology stack needed for a successful data lake is extensive and varied. This poses the question: how can enterprises possibly manage data across such a complex technology stack? Enter the data management platform. A robust data management platform is the key to enabling enterprises to manage and track data across various storage, compute and processing layers, as well as throughout its lifecycle. Not only does this transparency lend itself to reduced data preparation time, easier data discovery and faster business insights, it ensures enterprises can meet regulatory requirements around data privacy, security and governance.