By 2018, US will face a shortage of at least 140,000 data scientists. The numbers are from a McKinsey report about the frustration companies face when trying to hire these professionals. The demand keeps increasing as more and more companies look to leverage the value of data, so data scientist shortage won’t go away. This may not be a solution, but will sure help.
What are data scientists and why we need them?
Let’s take a step back and understand what exactly do these professionals, what are their skills and why the shortage.
Being a data scientist requires a comprehensive knowledge of a number of fields such as statistics, machine learning, data visualization, software development and techniques to manage and integrate data sets.
The type of tasks a data scientist can take are wide. In some companies they will be asked to work more as data analysts, producing visualizations and reports on company data from Excel, MySQL databases and others. In other organizations they can act more like a Data Engineer, setting up and managing data infrastructure or working with production code.
Ideally, a data scientist should also possess some industry knowledge to solve real business problems. As you can imagine such a combination of skills isn’t easy to find.
So what’s the solution to data scientist shortage?
The number of college degrees has increased significantly but it will always fall short on the demand. Companies could hire consultants and outside sources, but many aren’t willing to open their data to outsiders. Train their employees in data science could also be an option. Some organizations may argue that’s an expensive and slow option.
An other approach could be ease the burden on data scientists and IT teams. Normally, business users rely on those professionals to get even the most basic visualizations and reports. This creates a huge bottleneck.
With the right tools companies could delegate some of that tasks to business users. That’s when Modern BI comes to play.
Gartner defined Modern BI by saying: “The evolution and sophistication of the self-service data preparation and data discovery capabilities available in the market has shifted the focus of buyers in the BI and analytics platform market — toward easy-to-use tools that support a full range of analytic workflow capabilities and do not require significant involvement from IT to predefine data models upfront as a prerequisite to analysis.”
This includes capabilities such as Natural Language Query and Search, Self-Service Data Ingestion and Big Data Source Connectivity.
The goal is to shorten the time to insight, so users can answer more complex questions faster and this includes performing data modelling and ad-hoc queries without IT help or much training.
Although Modern BI wont increase the number of data scientist or even ease the hiring of those professionals at least it can help companies manage their analytics need and free data scientists to perform more complex tasks.
Do you think Modern BI can make a difference in data scientist shortage? How can companies solve this?