Riverside County IT Consolidation
With the great recession hit government over the past decade, many cities and counties have recovered, yet due to recently implemented pension reforms, many will not recover for the next 3-5 years. Up until this economic collapse, government was focused on service delivery and moving towards automated solutions faster than central technology organizations could respond. This created decentralized organizations with their own IT managers, focused on delivery of automated solutions to make departments efficient. However, when funding dried up, new measures required attention. Many elected officials and senior executives viewed IT as an expense, and many started exploring outsourcing of services and consolidation of IT services to realize savings.
In 2013, the Riverside County Board of Supervisors approved the replacement of a legacy phone system that was 30 years old with Cisco VoIP. The solution was to be implemented in 12 months to over 700 facilities across 7400 square miles. Departments were not very cooperative with the CEO vision of consolidation, and resisted centralizing their staff, so Technology Service Agreements (TSAs) were negotiated that merely allowed staff relocation.
After two years, the Cisco VoIP project was over budget, and only 50 percent deployed. Staff were centralized, but not shared, and the only savings that was realized was not filling vacant positions, so no efficiencies were gained and service levels declined. The central data centers remained larger absent of county servers since departments were not eager in moving their data centers over and were still willing to fund their decentralized environments.
In the decentralized environment, pushing up patches and updates was impossible
I started as CIO in May 2015. The consolidation included about 70 percent of the departments, but due to political barriers, seven departments were allowed to opt out (mostly elected officials who serve as department heads). In order to build trust with departments, I listened to their concerns, yet made sure I stayed the course of the CEO’s vision to bring success to the consolidation.
To get departmental buy in, it required strategies to slow down and to show wins before moving too fast. I was able to convince departments to work outside the TSAs and agree to consolidate help desk and field user technicians into a shared services model. It was initiated in November 2015, and on July 1, 2016, ServiceNow was rolled out as the IT Service Management system, eliminated over eight separate systems. A central help desk with now 24 staff all following the same processes has enable average waiting time from 9 minutes to less than 1 minute and handles over 600 calls per day with over 70 percent handled on the initial call. There is still room for improvement in customer service, and through training and customer surveys, that will improve in time. Over 70 field user technicians are spread across four regions and strategically located with a goal to solve all critical and high incidents within 2 hours. Mobile apps for creation and closure of tickets will help reach that goal in the next several months. Dashboards and metrics now drive our decisions and build credibility with departments rather than rumors and perceptions that ran rapid up to this point.
When the TSAs expired in July, I did not renew with departments, but established a standard Service Level Agreement that would apply uniformly to all departments. I also developed an annual strategic plan to clearly communicate the vision of the IT and the consolidation. More than $12 million were required for data center upgrades, help desk software and service, and hardware and storage for data center consolidation. With no funding available, I modified the rate structure in our charge back to be fixed by number of devices and servers and leased all software licenses and hardware over 5 years to keep rates low. While rates and budget went up about $2.5million/year, the new rate formula provides fairer equity to those that actually use our services. Therefore, there were some winners and losers, and the ones impacted are certainly vocal to elected officials. Most of the IT spend is still with the departments (about $120 million versus $90 million in the IT department). KPMG has been hired to provide recommendations on the external IT spend to increase efficiency and savings.
Our second phase of consolidation for this fiscal year is data center consolidation. A three year Enterprise License with VM-Ware will take our 2,300 servers from 30 percent virtualization to over 90 percent in year one, and within 18 months, 90 percent of our data centers will be consumed using VM-Wares NSX technology onto Dell blade servers and Compellent storage. Over 70 Systems are now centralized and shared. This enabled the consolidation of multiple active directory forests into one, and roll out of Office 365 will be completed by the first of the year, and a centralized on-line backup strategy utilizing EMC Networker is in process. In the decentralized environment, pushing up patches and updates was impossible, by now with the implementation of Microsoft SCCM, administration and control of the desktop will be realized within 12 months providing consistency in support and management across our desktops, laptops, and tablets.
Next up for consolidation are our Business Systems Analysts and Developers, which still work directly for the departments they serve. There are many single points of failures that will be resolved through the new-shared services model and the creation of primary and secondary support. Leveraging shared data base administrators, web masters, mobile developers, and application integrators, will enable IT to provide higher skilled resources rather that the generalist that existed in the past that try to become the one person that does it all.
The big question with this all is: Has it saved any money? So far, I have seen about $1.2 million in budgetary savings through attrition (not filling positions that are not critical). However rather than return that in savings in rates, there is further investment required in new ServiceNow Modules (Asset Management, Service Requests, Change Management) and Web Content Management and hosting, that will offset the savings, but will enable rates to remain flat. Since our County has no IT Capital program since everything has to be recovered through rates (due to state and federal reimbursement), the consolidation is not likely to produce savings due to requirement for technology refresh every year, but will enable much more efficiency and technology delivery/ innovation at little to no increase in costs. Lesson learned, never promise savings, only efficiency and cost avoidance, since IT is an investment and year over year growth continues. However, our CEO did make the right decision; he just needed a strategy that could be executed.