One Year Later, Observations on the Big Data Market

May 19th, 2016

Editor’s note: Welcome to Throwback Thursdays! Every third Thursday of the month, we feature a classic post from the earlier days of our company, gently updated as appropriate. We still find them helpful, and we think you will, too! The original version of this post can be found here.

Back in 2014, we discussed how the market looked like on our first birthday. As we hit three years, it seems like an appropriate time to look back on those observations. First, though, some thoughts on where things stand now.

We find ourselves transitioning from a market of early adopters, heading to one of the early majority in select industries, such as financial services and retail. As we teach about big data and data science at industry conferences, we meet an increasing number of enterprise leaders who are at the point of creating their strategy around data, and are innately aware of the importance of data as an asset.

There has never been more widespread awareness of the value of data science to business. The concept of artificial intelligence has again risen to the public consciousness: an identifiable way of explaining the role of data in creating value, rather than just analytics and reporting. Business leaders now have a way to understand that using data better involves changing processes, and the way they interact with customers, partners, and suppliers. IDC has forecast the market for big data technology and services to grow at a ~23% CAGR, reaching nearly $50B in 2019.

While we are moving towards broader market sophistication around data, there’s time yet before writing “data science apps” becomes an everyday reality for customers. Implementation is often still focused on infrastructure assembly as a first step; followed by development of customer applications that leverage data science capabilities. The software vendor landscape continues to show significant turbulence, with many similar players and messages, amplified by large investments and marketing budgets. Inevitably, we expect there to be rounds of consolidation, and for a few core enterprise data platform alternatives to emerge.

The most important first step remains developing a solid strategy in place. Data is only valuable if you do something with it. Our Data Strategy Position Paper has more details, and we’d love to hear from you in the comments on what you’re seeing in the market.

2014 Thoughts

We recently celebrated our first birthday here at Silicon Valley Data Science. Alongside building our own venture, we also believe that our clients should be creating Experimental Enterprises that adapt quickly to succeed.

We’ve had the good fortune to work with some amazing clients this year while providing our services. We have built data-driven capabilities that are core to the business for many of our customers.

We worked with an online retailer to develop their Data Strategy to provide them with a roadmap for future business and technology decisions. Through a collaborative effort between Silicon Valley Data Science and key stakeholders at the customer, we guided them on how to use data technologies to accelerate their growth plans.

Another customer — an innovative media & entertainment company — asked us to provide Architecture guidance on which analytical and data architectures to use to enable their Personalization agenda. We rapidly assessed their current state as well as their near-term aspirations to provide both an architectural plan.

We’ve also helped a variety of customers create new capabilities and develop products by deploying our data science and engineering teams in Agile Build teams. We’ve been able to rapidly develop software to help our customers working under tight timeframes and with highly productive teams.

Through all of the work that we’ve done and conversations we’ve had with clients, there are three big data observations that I can share on the big data market:

  • Techniques and approaches from one domain are quite often relevant in other domains. We think cross pollination is one of the best ways to innovate and get to analytic results faster.
  • Pursuing problems by placing blended teams of architects, data scientists, and engineers has led to better productivity and amazing results.
  • Our customers are asking the right questions about how data can be strategic for their business, not just trying to prove that they can deploy a new technology.

We are eager to help our customers build experimental enterprises that take the best of today’s technology to iterate and adapt their businesses for maximum impact. It’s not enough to just build a giant data repository, the question is how to build an effective platform that allows you to rapidly innovate with business-relevant applications.

We started Silicon Valley Data Science a year ago to teach people about our view on how to use data and ultimately help our customers with their data-driven aspirations. It’s been an incredible year of growth and opportunity, I’m looking forward to what we make happen in year two. Please feel free to contact us if you’d like to discuss your business and build an experimental enterprise.

Sign In