Sharing tips to reduce the cost of ownership of a corporate data warehouse by 30%
Challenge
Due to an influx of information, the system was no longer meeting the customer's needs. Initially, the warehouse collected information from a single source, but daily collection and processing took over 16 hours.
In addition, the client company manually collected and compiled data from three different sources to create reports used for managerial decision making. The task of creating the weekly reports was performed by three analysts over a three-day period.
Our team of IT specialists were tasked with:
- solving the performance problem of the existing historical data warehouse;
- design and implementation of a BI platform that would automatically receive operational data in a short time;
- reduction in the cost of ownership and maintenance of the existing storage (previously costing $100,000/year).
Solution
To solve this issue, our team designed and implemented a business intelligence (BI) platform to automatically receive operational data in a shorter time frame. We chose MPP GreenPlum distributed database as a historical data warehouse, which is an open-source and free software product. This allowed us to address the customer's concern about reducing the cost of ownership and maintenance of their existing solution.
The MPP Greenplum architecture distributes data between several servers, increasing the speed of data writing and reading. It also provides protection against data loss in case of server failure where the historical data storage is located.
After deploying MPP Greenplum, our team began adapting the existing data collection, transformation, and aggregation (ELT/ETL) processes. To provide visualization of analytical reporting, we chose Tableau, which was already enlisted under the “Leaders” category in the 4th Gartner quadrant at the start of the project.
The installed instance of Tableau-server was connected to the customer's Microsoft Active Directory authorization service, which allowed managing access rights to various reports in a flexible manner. It made possible the visualization of more than 50 different indicators using only 4 dashboards.
Technology
The final result
As a result of the implementation of the proposed solution for the source data warehouse and BI platform, the customer was able to:
- improve the performance of the data collection and processing system. The time spent on this compressor decreased from 16 hours to less than 2 hours;
- increase the performance of the data collection and processing system;
- reduce labor costs to ensure the data safety;
- increase the number of primary data sources from 1 to 3;
- improve the quality of data provided for analysis in 9% percent of cases,
- receive automatically generated reports in a few hours after the end of the reporting period, which previously took about three working days and three analysts to generate;
- reduce the cost of ownership of a complex solution with additional functionality by more than 30%.
Project implementation period: 9 months.
Contact us
Sales department
sales@blitz-brain.comMarketing department
Ready to discuss a project?
Tell us about your project in any form that is convenient for you, whether it is a clearly defined specification or a concept description.
.png){kind=small}