Despite the fact that the public sector has fallen behind private industry in embracing Big Data, recent survey results suggest that Big Data analytics are beginning to drive management decisions in government. In the recent GovLoop Guide, Innovations that Matter: Examining the Big Data Frontier, 73% of public-sector employees surveyed said that they use Big Data for budgeting and resource management. Another 60% said that Big Data improves control of waste, fraud and abuse. However, the survey also highlighted the many ongoing challenges of incorporating Big Data in the public sector. Following is a summary of highlighted challenges and best practices from the referenced guide.
Challenges
A major challenge for governments adopting Big Data is the often hefty capital investment required. Unfortunately, many government organizations have extended significant resources into existing IT infrastructures, which were in place well before Big Data was even on the scene.
Public agencies also tend to have existing data “sitting in virtual silos” without an established procedure for extraction. The issue of how to extract data is intensified by the expense of doing so, and the need to enlist a third party. Additionally, because government organizations store and monitor such large quantities of data, it is difficult to properly assess which data sets hold value.
Lastly, in terms of challenges facing government organizations adopting Big Data, a key issue is that the solutions available vary widely. While a specific solution may be ideal for one organization, it could be entirely wrong for another.
Best Practices
Considering the aforementioned challenges, it is crucial to examine best practices for government organizations adopting Big Data solutions.
Leadership
In order for adoption to take place, an organization needs to have open-minded leadership. Big Data cannot be undertaken without a holistic view of the organizational inter-dependencies and in certain cases, a drastic transformation must occur. This type of transition needs leadership that has the foresight for proper and effective Big Data integration.
Business Processes before Technology
It is important to focus on the mission and goals of the organization from a functional standpoint, as opposed to focusing on implementing the latest technology. Be aware of business requirements and source the technology to support those requirements.
Identify Specific Use Cases
Once Big Data objectives have been aligned with mission goals, identify a specific project to tackle. Start with a small initiative, outlining clearly defined performance metrics.
Leverage Existing Resources
Another best practice is to leverage the existing resources of your organization, then, augment resources as needed. If you have a well-established IT infrastructure, it may be relatively easily to transition to Big Data solutions, implementing system upgrades instead of a complete overhaul of the IT enterprise.
Plan for Legacy Integration
Many existing systems are simply not equipped for Big Data. For this reason, integrating legacy data systems can be difficult; specifically extracting data from the existing system. Also consider that these legacy systems are not typically made to handle data that is unstructured such as that from social media and video feeds. In this sense, augmenting the legacy system to cater to these types of data is a critical need. The best practice for integrating legacy systems is to give special consideration to planning an integration that will match fields within the system with relatively fast output processing.
Select Vendors Carefully
It is crucial that government organizations carefully select the right vendors and system architecture. Take the time to choose the right system for current needs, while also considering future requirements.
Consider Data Quality
Don’t lose sight of data quality and governance of the organization’s system. Effective data entry processes will ensure better quality data for analysis and decision making.
Train Many Employees – Not Just a Few
Lastly, make sure to build capacity within the organization for data accessibility, analysis and use. Train the entire organization on how Big Data will support the agency’s mission now and in capacities that are yet to be discovered.