The Brave New World of Big Data
by N Scheidegger

I was fortunate enough to speak at the recent Philanthropy Australia National Conference on a panel about Technology and Social Enterprise. The topic I focused on was data - or more specifically, Big Data – and I thought I’d share some of the thoughts I raised in my presentation.

We’ve been using data to help solve business problems for a number of years, but sometime in the late nineties, someone put the word BIG in front of it and suddenly it meant something slightly, meaningfully, different.

There are a lot of definitions of big data, but they all seem to converge on a few key points. Firstly, unlike ‘regular old data’, Big Data tends to come from more than one source or ‘data repository’. For example, combining POS purchase data with customer loyalty cards and online purchases, your local supermarket is able to get a fairly good picture of customer purchase behaviour.

Big data also tends to be unstructured – that is, the data doesn’t fit into neat tables with headings, and is rarely purely numbers based. Social media data is an excellent example of unstructured data, as you can imagine the sort of nonsensical acronyms, sentences, and concepts that make up a twitter feed.

A hospital in Washington DC provides us with a useful case study for this type of ‘soft’ data. The hospital analysed several years of anonymous data on patients – treatments, diagnosis, age etc. – to understand why patients were being re-admitted to hospitals. It wasn’t until they looked at patient discharge surveys that they spotted an interesting correlation: patients that tended to tick the box next to “feeling depressed” on their discharge form were more likely to be re-admitted. In other words, the mental health of the patient – not necessarily the physical health – may have more to do with re-admittance rates.

But what we’re doing with the insights from Big Data is more exciting still. One of most powerful uses of Big Data is in predictive analytics – that is, using the data not just to understand what is, but what will be. This isn’t about foretelling the future – although it can come eerily close to this. NCR – one of the larger suppliers of PoS and ATM systems in the US – has been spear-heading the use of Big Data in predictive modelling using telemetry data from field units. The company has gotten to the point where the units can now report back when parts are likely to fail in advance, allowing NCR to send replacement units or technicians to solve the problem before it even becomes a problem.

In the social enterprise sector, the results of Big Data are even more inspiring.

In Micro-financing, companies like RootCapital are using algorithms to understand where to invest based on a balance between financial and social metrics. As you can imagine, efficient allocation of resources in this context is complex. Root Capital are combining financial information with on the ground case studies that try to understand how one dollar on the ground impacts not just the business it was invested in, but also the community around it. Only in combining this hard and soft data can they really understand the true impact of a single loan.

In Australia, Data61 are pioneering the Big Data space. The core purpose of the group is to gather, cleanse and release a wide range of government data to benefit industry. They are currently working on using predictive analytics to try and solve the out-of-home care problem, using early intervention methods. 

Of course, there are a number of challenges facing the growing up-take of Big Data. The most obvious challenge is, of course, data privacy. This is a hot topic amongst policy makers, and something that all companies needs to be aware of. Sites like are helping not for profits be prepared for the compliance requirements of privacy, and even provides sample privacy statements and policies online. If you’re not sure if your business is compliant with applicable privacy laws, it’s important to seek advice in this area before you’re caught out.

The other issue is around capability – as data analysis can be expensive for the social sector and even small business. Platforms like Kaggle are breaking down these barriers, by offering a platform for organisations to post ‘competitions’ in the form of a data set and problem statement. Kaggle’s network of data scientists team up, enter the ‘competition’ and analyse the data to find solutions.

So where to next? Business looking to build their data capability should firstly look at what kind of data they collect, and have access to. You may be surprised by the kind of data available to the public, once you go hunting for it. If you have data and aren't sure what to do with it, talk to your IT team about what data analytics tools might be available to integrate with your systems.

Above all, create a culture that is serious about data - collecting it, securing it, and using it to drive meaningful business decisions.



There are no comments

Want to know more?

Call us

We're here to help! Phone to discuss our products, services and pricing options.

(02) 9869 5888

Let us call you

Get more information

Why choose 3rdmill?

The world of IT can be daunting and frustrating at times - but we are here to help make your life easier. We pride ourselves on being a one stop shop to fulfill your IT needs.

Read more