When it comes to building out analytics dashboards, Fawad Mohammed, head of data management at Catalyst, says many organisations encounter similar issues around the set up and development of underlying data warehouses. “Often, data engineers build warehouses from an input perspective, not an output perspective. They’re not thinking about the data analytics and the data science at the other end,” he says.
A common outcome is a mismatch between initial warehouse or analytics requirements and the actual result. To avoid that, organisations need to have data science or visualisation experts involved from the start, he says. “What does the business want out of the data warehouse? What do the data marts need to look like to ensure the analytics can be delivered efficiently without re-engineering the data model? Those things need consideration, and that’s why we look to hire data engineers with a visualisation skillset too. Otherwise, you end up with a data model either missing requirements or it won’t have been designed to allow you to put a Power BI or a Qlik dashboard directly on top of it.”
Stay agile
Of course, total foresight of requirements may well be a practical impossibility, and so Mohammed stresses the importance of ensuring data warehouse construction can be agile and react rapidly to changing requirements. He cites the scenario of a new director or head of department coming into a business and wanting a different style of reporting. “The original requirements are now gone, and the warehouse needs to be architected and delivered to meet the requirements of that individual, which will take months.”
That’s often too slow in the current business world, he says, where requirements may change on a weekly, or even daily, basis. And, even once those changes have been implemented, requirements could still shift, he says. “The first iteration of many data warehouses are successful, but the sheer amount of time it takes to remodel it will mean that, unless changes can be delivered rapidly, it’s almost pointless.”
“The first iteration of many data warehouses are successful, but the sheer amount of time it takes to remodel it will mean that, unless changes can be delivered rapidly, it’s almost pointless.”
Fawad Mohammed, head of data management, Catalyst BI
One way Catalyst BI has approached the issue is to build what Mohammed calls a “tactical solution”, which involves taking the business logic for a dashboard and building it in the analytics toolset first. “Once those dashboards are live, we can either take the business logic and build the data warehouse off the back of that – if there isn’t one already – or move the tactical data model into the warehouse. That’s an offline exercise that takes just a couple of months, but the end user already has their dashboard,” he says.
Automatic code
Though it may seem this approach represents additional effort, Mohammed explains that it’s more sustainable in the long run for many organisations, as they otherwise run the risk of utilising inadequate or inaccurate reporting. He cites as an example a large retail organisation that Catalyst BI works with, which found it had millions of pounds worth of dead stock not identified by the data warehouse. As a tactical solution, Catalyst BI identified the missing dead stock data and created a tactical data model within the analytics solution to surface that information within a week.
This scenario, he adds, is commonly the result of businesses struggling to balance the resourcing requirements of both developing a new data warehouse and servicing an existing one. “Organisations can end up with huge amounts of ‘technical debt’ – stopgap measures or uncodified architecture – that they keep building on top of. You can end up spending weeks unpicking code, trying to understand how something was implemented three or four years ago – that’s not sustainable. We’ve seen one organisation find their original reporting figures were out by 15% after they reworked it, because there was very little understanding of what had already been implemented previously.”
“Organisations can end up with huge amounts of ‘technical debt’ – stopgap measures or uncodified architecture – that they keep building on top of. You can end up spending weeks unpicking code, trying to understand how something was implemented three or four years ago – that’s not sustainable.”
Fawad Mohammed, head of data management, Catalyst BI
Another solution to the above issues, Mohammed says, is to automate the implementation of building a data warehouse, accelerating the process and making data structures more visible. “Compared with the conventional approach that takes multiple data engineers and a data architect 12 to 18 months to complete, Qlik Data Integration (QDI) automates the development of a data warehouse or data lake. It means you can build a data warehouse from source all the way out to analytics tools like Power BI, Qlik and Tableau, or machine learning tools such as Data Robot within a matter of months,” he explains.
Further, automating code creation means changes to data warehouses can be quickly switched out as requirements change. “If a new CIO comes into the organisation and wants to change the data warehouse to another provider, you’d normally have to start writing code from scratch. With QDI, you can just change the dropdown to deploy the data warehouse into a new cloud and it writes the code for you – whether that’s a Snowflake, Google BigQuery or Azure Synapse environment – within hours. The data is obtained in real time because the technology is doing the coding – no other technology can do that.”