Data: What Industry Wants
August 22nd, 2013
What should you do with data? That’s the overwhelming question facing industry today. Since launching Silicon Valley Data Science we’ve had many conversations with customers, and we hear three issues more often than anything else: data overload, choosing analytic approaches, and having the right skills and resources.
Industry faces a broad spectrum of data challenges. The increased profile of data in business means many want to know what big data really is, or if it’s necessary to hire a data scientist. Many of our customers understand the potential data has for their business, and have ambitions such as transforming legacy data architectures, or ensuring that new product development happens in a data-driven environment.
Over all industry sectors, we’ve found that companies tend to have strategic data issues, ones that materially affect the business, requiring both long-term planning and near-term action. Today’s business leaders are strongly aware that they can use their data better, and often have a strong idea of the benefits this would bring to the business.
There are three data challenges we hear more than others.
- We can’t capture the data we’ve got. While big data advocates tout the advantages that adding new data sources bring, many companies can’t yet exploit the data they’re generating already. They’re not yet at the stage where data can bring them a total picture of their business, and are having to throw away data from existing systems.
- What analytical approaches should we use? Capturing data is a big part of the battle, but the analytics use cases serve business needs and drive the underlying architecture. How can analytics be applied to existing systems, and how should new systems be architected to ensure we can fully take advantage of analytics?
- We don’t have the skills. Many companies don’t have the skills to evaluate new data technologies and understand how they would address today’s business problems, or to plan for tomorrow’s needs.
So what should an aspiring data-driven business do to take steps forward? It’s a problem we’ve been thinking hard about as we assist industry in realizing data ambitions.
We approach data-driven opportunities by first educating the business on the art of what’s possible, often a transformative stage as eyes are opened to the potential data can bring. We then move to a planning stage, identifying use cases that can demonstrate business or technical benefit. The best results always come from addressing a real need first. Finally, we use agile methods to quickly prove the viability of high value use cases, and ultimately, move the new data capabilities through the pilot stage and into production.
Agile, cross-functional teams
The core of our approach is to field teams that include business domain expertise, data scientists and engineers to create data-driven applications by iterating rapidly. In recent years software companies have embraced the art of experimentation, using agile approaches to evolve requirements and product through iterative development. A similar approach works well for addressing data and analytic challenges.
Our agile approach opens up a number of ways to deal with the three main data challenges mentioned earlier.
When faced with overwhelming data, look for a part of the business that would benefit from having more data. Once a few candidate use cases have been identified, it’s much easier to envision which team, which data, and which technology can be used for rapid experimentation.
Quickly building out a well-defined vertical use case with new data—from data architecture through consumption to analytics—is an excellent way to answer questions about what analytical approaches to use. By building out a narrow use case that validates critical parameters for future planning, we’ve worked with customers to quickly assess not only a new data source, but also a candidate data architecture.
Lastly, bridging a gap in skills requires bringing in outside help. Especially at the early, exploratory stages, you want a team that brings an agile and collaborative approach to solution development, while also bringing both data science and engineering skills. Today, experimentation is cheaper and faster than ever, thanks to the flexibility of cloud-based software and platform services, such as Amazon’s AWS. Pair that with the right partner, and you’re well on the way to driving your business further with data.
