Big Data specialist training and report automation: How we helped a university create an information processing complex

Case study: Education

Challenge

As part of this project, we are collaborating with one of the leading technical universities with over 19,000 students. It specializes on providing high-quality and competitive education and research services.
The university has its own information system that ensures an efficient learning process. This system stores all the data about students, teachers, scholarships, salaries, and classes. Like any other educational institution, our client regularly provides a multitude of statistical forms and reports to the higher organization. Therefore they approached us to automate this process to eliminate human errors and decrease time for report preparation.
In addition, the university aimed to take a leading position among educational institutions specializing in training Big Data professionals and to improve its rating among other universities. So to get a professional consultation and knowledge in the field of Big Data was another objective of our client.
Our client had three main goals:
  • Ensure the operational activities of the university by automating the process of preparing and providing statistical reports to the corresponding ministry.
  • Enrich their own knowledge and become a leading university in the field of Big Data in their region.
  • Start training their own Big Data specialists.
To achieve these goals, we split them into the following tasks:
  • Develop and implement a unified corporate data warehouse for collecting, processing, storing, analyzing, and visualizing information circulating in existing information systems.
  • Develop reports for visualizing, and providing statistical information with data extraction and filtering.
  • Create a training program for mastering practical skills in Big Data technologies.
  • Prepare methodological materials (lectures, practical classes) in the Big Data area and transfer knowledge in this area to the client's teachers.
The corporate data warehouse was expected to provide high availability and performance in data processing, confidentiality, integrity of processed and stored information, horizontal scalability, and serve as a technological basis for Big Data.

Solution

From the start, our specialists designed the structure of the university's data management system for 3000 users of different levels. Then, as planned, the team also developed and implemented reports for the ministry that included statistical data on the number of students enrolled, practicing teachers, support staff, provided scholarships, salaries, organization expenses, and other operational information.
Then our team focused on creating an educational program that would allow the university to acquire recognition in the Big Data field. We helped the university gain an advantage over graduates from other universities by not only teaching students what Big Data is but also providing them with practical training on Big Data tools to ensure the engagement and comprehension of both staff and students.
Initially, we agreed with the client on the educational program but during the project, our specialists who closely collaborated with university professors realized that in addition to lectures, practical exercises were also necessary for the implementation of the methodological part of the educational program. As a result, we provided students with a full set of both theoretical learning materials and practical tools for working with Big Data.
So far we have transferred knowledge and experience to the client on how to use the solution, which will allow the university to independently manage the comprehensive data management solution.

Technologies

Ansible Apache Airflow Apache Atlas Apache Kafka Apache Superset Birt Grafana Greenplum Liquibase Report Server

The final results

We designed the structure of the university's Data Warehouse, set up processes for loading and transforming data from the existing information systems, developed statistical reports with data filtering and extraction based on the required attributes, created a lecture course for the university, and deployed a training program for practicing Big Data tools.
Benefits for the Client:
  • The Client decreased time spent on data collection by the automatic generation of a large number of reports. The processes also became more transparent due to the visual representation of the results.
  • The Client gained new data management capabilities by using the Data Warehouse.
  • The university is now training highly qualified graduates by giving theoretical and practical lessons on Big Data and data management, thus increasing its own chances of taking a leading position among universities in its region.
Project implementation period: 2 months.