Information overload or opportunity? – what is big data?

An earlier blog post highlighted previous examples of businesses underestimating the rate and impact of technological developments. The ‘Internet of Things’ (IoT) and Big Data are terms that are increasingly mentioned in connection with business development. Are these of universal relevance?, or matters that chiefly concern larger, more complex companies?. We expect that the developments will come to affect all….

Smaller and faster data processing components give new opportunities for sensors that provide information about industrial processes. This is expected to give a second ‘Information Revolution’ using this ‘Internet of Things’ where sensors can send, receive and react to information flow. This leads to the term ‘Big Data’ which describes the large about of electronic information that is becoming available, and how it can be analysed computationally to reveal patterns, trends and inter-connectivities between the measurements.

Interconnectivities can be identified by use of a technique called ‘Principal Component Analysis’, which is a statistical method analysing the variations in variables over time, and finding which variables tend to affect, or be affected by others. From a set of apparently randomly varying variables the technique can reveal, for example, that product quality parameters are adversely affected when a certain component becomes worn, unless certain other variables are manipulated to compensated. Interpretation from humans is still needed, but increasingly the information is being used directly by machinery, in what is termed ‘Machine Learning’.  The machine gathers information from the interconnected sensors, identifies trends and influences and adjusts the operation accordingly. This can be both faster than having humans review the data, as well as potentially more powerful, in terms of the amount of information that can be processed. Of course, there is also potential for mistakes.

The application of big data and machine learning

The application of big data and machine learning is also an opportunity for suppliers to improve their products or services with additional features. An earlier blog article mentioned Karsten Moholt AS in Norway who gather process data to guide preventative maintenance decisions. Another company, Xero, based in New Zealand has developed an accounting software system that analyses invoices from a million customers, mainly SME businesses, and gives predictive assistance for data entry for accounting software, attaching account codes to financial records. Customers host their accounts on a cloud-based server, and Xero analyses trends in the anonymised data that allows the software to be more efficient and accurate, giving savings in data entry costs for the companies that use it. An accuracy of 80% after 4 invoices and 90% after 50 invoices is claimed. However, experience with implementing the system revealed greater variety than expected in what code different companies filed items under. Nevertheless, the company is continuing to improve the machine learning capability of the software algorithm.

A different illustration of the application of ‘Big Data’ is the suggestion that data mining of social media was used to target campaigning elections, and was used by the winning sides in the Brexit and recent US President elections. A company, Cambridge Analytics, offers analytical services for modelling target audience groups and predict behaviour. The method apparently originated from work published on predicting private traits and attributes from analysing social media activity. It seems that ‘Big Data’ and machine learning is likely to affect us all, whether we are aware of it or not…

Ross Wakelin

Northern Research Institute Narvik A.S., ross@tek.norut.no, (47) 99 252 485

kommentointi suljettu.