PixieMe/Shutterstock

Public Well being England has admitted that 16,000 confirmed coronavirus instances within the UK had been missed from each day figures being reported between September 25 and October 2. The lacking figures had been subsequently added to the each day totals, however given the significance of those numbers for monitoring the outbreak and making key selections, the outcomes of the error are far-reaching.

Not solely does it result in underestimating the size of coronavirus within the UK, however maybe extra necessary is the following delay in coming into the main points of constructive instances into the NHS Check and Hint system which is utilized by a workforce of contact tracers. Though all those that examined constructive had been knowledgeable of their outcomes, different individuals in shut contact with them and probably liable to publicity weren’t instantly adopted up (ideally inside 48 hours). This was a critical error. What might have precipitated it?

It emerged later that that day a “technical glitch” was accountable. To be extra particular, the lab check outcomes had been being transferred to Excel templates. The templates hit a restrict within the variety of rows they might deal with after which did not replace with extra instances added. The difficulty was resolved with all new instances added to the totals reported over the weekend by breaking the information down throughout smaller spreadsheets.

The difficulty could have been fastened, however individuals’s confidence within the testing system in place in England will undoubtedly take a knock. It’s additionally doubtless that politicians and media will use this as political ammunition to argue the incompetence of presidency and Public Well being England. Is that this the suitable response? What ought to we take away from this error?

An avoidable mistake

We should always not overlook that the federal government and public well being employees are doing an extremely difficult and demanding job coping with a pandemic. However this sort of mistake was avoidable. We stay in a world of huge information, with synthetic intelligence and machine studying permeating all facets of our lives. We now have good factories and good cities; now we have self-driving vehicles and machines skilled to exhibit human intelligence. And but Public Well being England used Microsoft Excel as an middleman to handle a big quantity of delicate information. And herein lies the issue.

Though Excel is in style and generally used for evaluation, it has a number of limitations that make it unsuitable for giant quantities of knowledge and extra subtle analyses.

The businesses that analysed the swab exams to establish who had the virus submitted their outcomes as comma-separated textual content recordsdata to PHE. These had been then ingested into Excel templates to be uploaded to a central system to be made out there to the Check and Hint workforce and authorities. Though right now’s Excel spreadsheets can deal with 1,048,576 rows and 16,384 columns, builders at PHE used an older Excel file format (XLS as a substitute of XLSX) leading to every template with the ability to retailer solely round 65,000 rows of knowledge (or round 1,400 instances). When the restrict was reached, any additional instances had been left off the template and due to this fact constructive instances of coronavirus had been missed within the each day reporting.

The larger challenge is that, in mild of the data-driven and technologically superior age during which we stay, {that a} system primarily based on transport round Excel templates was even deemed appropriate within the first place. Knowledge engineers have for a very long time been supporting companies with managing, reworking and serving up information, and growing strategies for constructing environment friendly, sturdy and correct information pipelines. Knowledge professionals have additionally developed approaches to data governance, together with assessing information high quality and growing applicable safety protocols.

For this sort of customized software there are many information administration applied sciences that would have been used, starting from on-site to cloud-based options that may scale and supply managed information storage for subsequent reporting and evaluation. The Public Well being England builders little doubt had some purpose to rework the textual content recordsdata into Excel templates, presumably to suit with legacy IT techniques. However avoiding Excel collectively and transport the information from supply (with applicable cleansing and checks) into the system would have been higher and decreased the variety of steps within the pipeline.

The blame sport

Regardless of the advantages and widespread use of utilizing Excel, it isn’t at all times the suitable software for the job, particularly for a data-driven system with such an necessary perform. You may’t precisely report, mannequin or make selections on inaccurate or poor high quality information.

Throughout this pandemic we’re all on a journey of discovery. Moderately than level the finger and play the blame sport, we have to replicate and study from our errors. From this incident, we have to work on getting the fundamentals proper – and that features sturdy information administration. Maybe relatively regarding are experiences that Public Well being England is now breaking the lab information into smaller batches to create a bigger variety of Excel templates. This appears a poor repair and doesn’t actually get to the basis of the issue – the necessity for a strong information administration infrastructure.

It’s also outstanding how shortly expertise or the algorithm is blamed (particularly by politicians), however herein lies one other elementary challenge – accountability and taking accountability. Within the face of a pandemic we have to work collectively, take accountability, and deal with information appropriately.

Paul Clough works part-time for Peak Indicators, a UK-based Enterprise Intelligence & Analytics firm.