Like most complex organizations today, Minnesota state government relies on numerous computer systems to support its internal administrative functions. Recognizing the need to periodically upgrade these systems to meet the changing needs of government, the state initiated the "Statewide Systems Project" (SSP) in 1991. Its purpose, as defined in 1992, was to replace the state's computer systems for accounting, procurement, and payroll functions, and create new human resources and decision support systems.
The project was sponsored by the departments of Finance, Employee Relations, and Administration, which assembled a project development team of state employees and selected outside consultants to assist with different phases of the project. By the time the systems became operational, between mid-1995 and early 1996, the project cost nearly $36 million--over 50 percent more than anticipated when the project was originally conceived in 1991.
The higher than anticipated costs, and the complaints of some users that the new systems were inadequate and hard to use, prompted the Legislative Audit Commission to authorize an evaluation of the Statewide Systems Project. The commission wanted the evaluation to examine the expenditure of funds and to determine whether, on balance, the project has been successful. The commission also thought that a review of the state's experience with the Statewide Systems Project and similar large computer development projects could help policy makers make better decisions about future systems projects.
Our evaluation addressed these key issues:
What are the costs and benefits of the new computer systems to date? Do the new systems meet the state's planned objectives? To what extent have the new systems saved the state money?
How satisfied are the users of the new systems?
What steps should be taken now to address current problems, and what strategies should the state follow to maximize the chance of success with future computer development projects?
To answer these questions, we conducted interviews with more than 120 staff in 30 state agencies about the project's development and implementation and surveyed a random sample of users of the new systems. We also consulted with national experts and reviewed the literature on systems development. In order to provide a broader context for our analysis, we also briefly reviewed five other large systems projects developed in Minnesota recently.
We qualify our conclusion and call the Statewide Systems Project "moderately" successful because numerous problems, including higher costs, resulted from the project's extensive scope and complexity. In addition, many of the benefits anticipated from the new systems, such as cost savings and enhanced functionality in some areas, have not yet materialized. The new procurement system has additional shortcomings and needs to be re-examined.
On the other hand, we think these problems should be put in context. Developing new computer systems is a complex and difficult activity, and Minnesota's experience with SSP has been more positive than many similar efforts in both the private and public sectors. A leading consultant told us that 80 percent of systems projects "fail" because they are not completed on time or within budget, or do not meet user expectations.
Overview of the Statewide Systems Project
The Statewide Systems Project consists of three major components (see Figure 1):
The new accounting system is known as the Government Financial System (GFS).
The new procurement system is called the Advanced Government Purchasing System (AGPS).
The first two systems are interrelated and are frequently referred to as the Minnesota Accounting and Procurement Systems (MAPS). In addition, the project developed an "information warehouse" which brings together data from the new systems and enables users to generate reports for decision support.
For most of its existence, the Statewide Systems Project has been directed by a steering committee, consisting of the commissioners of the three sponsoring agencies and deputy or assistant commissioners from four other large agencies. The Department of Finance received appropriations of $300,000 in 1991 and another $1.8 million in 1992 to help the project steering committee, assisted by a consultant and five work groups, plan for the new systems. The steering committee decided to purchase and modify existing software packages as the basis for the new systems and hired a major national consulting firm, Andersen Consulting, to customize them to meet the state's needs.
Computer System Development and Operating Costs
The cost estimates for the Statewide Systems Project grew as the project developed. The earliest rough estimates in 1991 called for a $15-20 million project to replace the accounting and payroll systems. An estimate of $19.5 million was made in the steering committee's 1992 Report to the Legislature to replace the accounting, payroll, and procurement systems and add a new human resources and decision support system, but when the project received bids from vendors in March 1993, the projected budget rose to $26.1 million. By 1996, the total cost of the Statewide Systems Project exceeded $35.8 million. This does not include the costs of training, networking, state employee release time, or most equipment that needed to be purchased by individual agencies in order to use the new systems.
According to our study:
Changes were needed in part because several separate agency accounting systems that were to be replaced were not identified until late 1993. In addition, it took much more effort than originally anticipated by either the state or the consultant to modify all components to meet the state's requirements.
Between March 1993 and mid-1996, the project's total cost rose almost $10 million. Approximately $5 million of this amount went to Andersen Consulting for the additional work required by the changes in the systems' specifications. In addition, about $1.6 million was spent on additional support for agencies, about $2 million went to pay state employees for work that was originally unanticipated, and the rest was spent on computer usage charges to test the new systems.
Over time, the most important computer system costs are those required to operate the system from year to year. We found that:
The project's earliest operating cost estimates noted that the four new systems would cost more than the old systems--perhaps as much as 250 percent more than the $1.7 million it took to run the old statewide accounting and personnel/payroll systems--because they would handle more transactions and perform additional functions. The 1992 Legislature was told by the then Commissioner of Finance that operating costs would be less than $5 million per year. However, we estimate that operating costs will exceed $16 million in fiscal year 1997. This includes the costs for approximately 60 staff in the Department of Finance's "Mn-ASSIST" office who are responsible for making technical modifications to the systems and providing user support. <FN - Additional support is provided by functional analysts in the sponsoring agencies.> According to agency managers we interviewed, even this level of staffing is inadequate to meet current needs, and therefore operating costs may rise even further in the future.
The state has made a number of efforts to reduce operating costs. Sponsoring agencies' staff have continually optimized the underlying computer code and data to make the system consume fewer mainframe computer resources. In addition, they have evaluated and decided to implement a non-mainframe solution for the information warehouse. According to the sponsoring agencies, this solution is projected to save over $1 million per year in operating costs. Non-mainframe systems offer the potential of saving money on operating costs (and possibly improving response time) for other components of the Statewide Systems Project. We think that the sponsoring agencies should continue to explore non-mainframe solutions for the other components of the Statewide Systems Project.
Finally, the accounting software that Minnesota chose does not accommodate transactions with four-digit dates, nor did any of the accounting packages offered for purchase by the state in 1992. This issue--often referred to as the "Year 2000 problem"--was discussed when the vendor was chosen, and although the steering committee realized it would have to be fixed later, they decided to continue with the software acquisition. According to project managers, there was no attempt to keep this information from the Legislature. However, the steering committee's action effectively committed the state to additional expenditures, and the Legislature was not informed of the problem until January 1995. Therefore, the Legislature did not have all of the relevant facts when deliberating on the 1993 and 1994 appropriation requests. The accounting and procurement software now in use will have to be upgraded at a cost estimated to be about $4.5 million. <FN - The procurement software technically accommodates the year 2000, but it does not accommodate four-digit dates, and because it works in concert with the accounting system, needs to be upgraded.>
Achievement of Objectives
Early on, the proponents of the Statewide Systems Project emphasized the many benefits that would result from its implementation. We found that, in fact:
For example, the new systems collect more information than the systems they replaced. Also, information is believed to be more widely available, easier to obtain, more timely, and thus, in some cases, more accurate. In addition, for some components, the new systems give users the ability to generate custom reports on demand. The greater access to information--and the greater ease of manipulating it--has enabled agencies to better oversee accounts and has increased their ability to use information for planning purposes.
The Statewide Systems Project also forced agencies to improve their agency computer systems and networks. Some agency employees said that increased interconnectivity of state agencies has enabled better communication and cooperation between agencies. In addition, the user groups for the new systems have increased interagency communication.
The new accounting system (GFS) has many new features that were not available before and gives a more accurate picture of the state's financial status. Agencies now have accounting information on line, and, unlike the old system, the new system has largely eliminated the double-entry of information and eased the difficulty in reconciling accounts.
The new procurement system (AGPS) has enabled state agencies to decentralize the procurement function in some cases. Although on average purchase orders take longer to enter, overall, those that are entered successfully take less time to process since orders no longer have to be sent to accounting. Also, all procurement transactions are now handled by one computer-based system.
Finally, the new human resources/payroll system (SEMA4) is a major step forward since the state's human resources function is now computer-based for the first time. The new system is relatively easy to learn and navigate, and on-line processing gives agencies more time to complete certain tasks. Under the new system it is easier to transfer employees between agencies, record salary changes, and hire people much later in the pay period.
According to the state's consultant and project sponsors, the project was supposed to: 1) save money by restructuring financial and compliance control processes and by replacing paper-based transactions with electronic processes; 2) provide better and more accessible information to policy makers and administrators; 3) improve the linkage between statewide financial and human resources information; 4) replace numerous separate computerized accounting systems used by individual agencies; 5) provide flexible systems that would be easy to change to meet future needs; and 6) provide better service to the state's citizens, vendors, and customers. <FN - KPMG Peat Marwick, Minnesota Statewide Systems Project: Report to the Legislature (Minneapolis, February 25,1992), 2-4.>
We examined each of these specific objectives to determine the extent to which they have been achieved so far. Unfortunately, our examination was complicated by the lack of baseline information about the old computer systems and the difficulty of quantifying many of the measures of system performance. Nevertheless, it is evident that many of the original objectives have not been met. Moreover, we conclude that many of the original objectives were probably unrealistic. The ambitious objectives heightened expectations among policy makers and contributed significantly to user disappointment and dissatisfaction.
Despite the gains, the Statewide Systems Project has not achieved all of the specific objectives it set out to accomplish. We found that:
The new systems have moved much more information on line than was previously available, but many users find the "information warehouse" hard to use.
Procurement system users are concerned that the system's complexity discourages them from entering data as intended and results in lower data quality than planned.
The accounting and procurement systems have not been seamlessly integrated as expected.
Out of 14 separate accounting systems maintained by individual agencies and targeted for elimination, 6 are still operating.
In addition, the early proponents of the Statewide Systems Project predicted that significant dollar savings would accrue from the newly designed computer systems. Specifically, they said the project would 1) improve collection of accounts receivable by about $1 million per year; 2) reduce the price of commodity purchases by up to $2 million per year; 3) achieve a one-time savings of $7-14 million by eliminating the 14 individual agency accounting systems; 4) save $2.8-3.5 million annually by eliminating paperwork; and 5) save $16 million annually through business process "re-engineering" or redesign. The sponsoring agencies note that there were additional re-engineering ideas incorporated into the project's design for which the benefits were never estimated.
Unfortunately, there is no readily available method to determine whether these projections have been realized, and we found that the projections themselves were not calculated rigorously. Nevertheless, we think there have been some cost savings in the targeted areas, but less than projected. We conclude this because we found that:
Fewer state agencies are using the accounts receivable system than anticipated--currently 10 state agencies use the system.
The Department of Administration has not yet used information from the system to negotiate commodity contracts, and, therefore, it would be difficult to attribute any savings to the procurement system yet.
The state may have realized savings of $4-8 million from eliminating agency accounting systems, roughly half the amount projected.
To the extent that savings have materialized, they tend to offset the increased system operating costs we noted earlier. But because there is little measurement of the benefits claimed for systems projects after they are completed, we recommend that:
In our view, the proponents of the Statewide Systems Project oversold the benefits that could realistically be expected from the project and set the expectations of policy makers and users too high. While there have been significant gains from the implementation of the new systems, the overall costs are higher than expected, the benefits are lower, and many expected outcomes have not been realized.
Some objectives might be met in the future if, for example, the EDI (electronic data interchange) module of the procurement system and the workers' compensation, recruitment, scheduling, and training modules of the human resources system are implemented. These modules would significantly reduce the need for paper documentation for many transactions. The Department of Administration plans to start a pilot test of the EDI subcomponent of the procurement system in January 1997, and the Department of Employee Relations has plans to implement the workers' compensation and training modules shortly thereafter.
Many of the enhancements to the system necessary for non-sponsoring agencies to fully use the system are still on the development "wish list." We think that continual investment should be made in the systems in order to increase their functionality and increase future benefits. The sponsoring agencies should periodically assess needed improvements and report to the Legislature.
User Experiences with the New Systems
The users of the new computer systems--thousands of employees working for state agencies at many locations around Minnesota--have now had over a year of hands-on, practical experience working with the new systems. Their judgments about the success of the new systems, and their observations about problems and possible solutions, are important. Accordingly, we surveyed 459 users of the statewide systems, asking them how satisfied they were with various features, such as system response time and operating hours. We also asked them whether their opinions had changed over the last year. To supplement our sample of users, we also conducted interviews with over 120 staff in more than 30 state agencies.
Overall, we found mixed levels of satisfaction with the new systems. Users of all the systems were happy with many features, including the ability to conduct on-line inquiries and the service they received from the Mn-ASSIST office. On the other hand, users were generally dissatisfied with the standard reports and the difficulty of generating reports from the information warehouse. They were also unconvinced that the new systems have saved staff time, money, or the use of paper.
Of course, the experience of users, and their degree of satisfaction, depends largely on which system(s) they have used. We found that:
Human Resources and Payroll
Survey responses and interview comments from the human resources professionals we interviewed were generally positive about the human resources system (SEMA4). Respondents were pleased with many features of the new system, including the ease of making inquiries (76 percent), being able to process transactions on line, not having to send paper forms to the Department of Employee Relations for processing, and current advisories and special reports (66 percent). Users were less satisfied with the standard reports (45 percent satisfied) and the information warehouse (only 33 percent satisfied and 25 percent dissatisfied).
Both satisfied and dissatisfied users reported that:
Nearly two-thirds of all human resources users rated system response time as a problem. Some actions, during some time periods, can take several minutes to process. The sponsoring agencies are actively assessing where bottlenecks exist in the system.
The limited availability of the system to process some types of human resources transactions is another problem cited by about one-fourth of users. Some transactions can only be processed during a few days of the 10-day payroll cycle. The sponsoring agencies have responded to this problem and expect to complete a project at the beginning of 1997 that will permit transaction processing during 7 of the 10 days in a payroll cycle.
Users were also generally satisfied with the payroll component of SEMA4. Users were satisfied with their ability to complete transactions (75 percent), navigate the system (72 percent), and make inquiries (67 percent). Users were also happy with the ability to have an on-line history of payroll, fewer errors in payroll because of edits in the on-line entry, and on-line business expense reporting. Users were somewhat less satisfied with the on-line help feature (47 percent) and the standard reports (52 percent). Like human resources system users, 73 percent of payroll users told us they were unhappy with the system's response time, and 21 percent reported difficulty in generating reports from the information warehouse. We heard similar assessments in our personal interviews.
Accounting and Procurement
As noted above, users were less happy with the accounting and procurement components of the Statewide Systems Project. Overall, 41 percent of accounting system (GFS) users said they were satisfied, 37 percent were dissatisfied, and 22 percent were uncertain. The accounting users liked the ability to complete transactions on line and to perform on-line inquiries. Users also frequently mentioned that they liked having more information available. Users were generally satisfied with their ability to complete transactions (54 percent), navigate the system (52 percent), and make inquiries (52 percent). Users were much less satisfied with the standard reports (33 percent). Also, two-thirds of the users of the new accounting system told us that it does not perform all of the functions of the old system, and users thought that the new system does not save staff time.
Users were least satisfied with the new procurement system (AGPS). Overall, only 35 percent of users said they were satisfied, while 40 percent were dissatisfied and another 25 percent were uncertain. Users were divided about their ability to complete a transaction (46 percent satisfied), navigate around the system (45 percent satisfied) and make inquiries (46 percent satisfied). We found the highest level of dissatisfaction with the procurement system in our interviews and survey comments from state supervisors and managers.
A significant number of users said that completing transactions requires navigating through too many computer screens and that the system employs too many complicated document types. <FN - Document types are the equivalent of different types of electronic forms used to enter information into the system.> The general consensus of state managers was that the state was trying to collect too much information. We were told that some agencies such as the Department of Transportation do need to keep track of item level data for inventory purposes, but most agencies do not. Also, the "seamless integration" between the accounting and purchasing systems has not occurred. Finally, users were least satisfied with the contents of the AGPS standard reports (27 percent satisfied, 35 percent dissatisfied). In part, this is because procurement information has not yet been put into the information warehouse and only a limited number of standard procurement reports were ever programmed.
Managers in several agencies, including some of the most frequent users, told us that they would like to abandon the new system altogether and use the new accounting system, with modifications, to conduct procurement transactions. Many users told us that they were taking shortcuts around AGPS in order to get their business done. <FN - The sponsoring agencies note that they have approved some of these "workarounds."> As a result, the data in the system are incomplete and unreliable, and one of the primary benefits claimed for the system, that the state can negotiate better contracts using the information gathered by AGPS, has not yet occurred.
The Department of Administration is aware of these problems and received high marks from users for attempting to solve the problems, especially in recent months. However, at a minimum, we think the department should try to reduce the numbers of document types and make it easier to navigate through the system. In addition, we recommend that:
Alternatives include 1) modifying AGPS to make it work in concert with agency business needs, 2) making the system optional for certain transaction types and/or for certain agencies, or 3) replacing AGPS with an alternative system.
We acknowledge that the state needs a central procurement system and that replacing AGPS would be an expensive and time-consuming (and, therefore, not the most attractive) option. But, we still think there would be value in a "ground zero" review. We also think an independent consultant may need to be hired to help the user agencies and the departments of Administration, Finance, and Employee Relations evaluate the costs and benefits of state agencies' current use of the AGPS system. The basic question is whether or not the state really needs the massive amount of data that it currently is collecting through its new procurement computer system.
A separate issue involving the procurement system relates to the collection of sales taxes on purchases by state agencies. Agencies have been liable for paying the sales tax since 1987. One of the "re-engineering" ideas implemented by the Statewide Systems Project was to have the tax paid directly into the state treasury instead of giving it to the vendor who would then pay it back to the state. Now the only way for a state agency to make a purchase that is taxable is to use AGPS. But all the managers we interviewed said that this has proved time-consuming and costly for state employees. Basically, they report, vendors know more about the taxability of the products they sell than state employees do.
But if the procurement (AGPS) system were replaced or made optional, some way would have to be found to facilitate the collection of sales taxes from state agencies. Currently the new accounting system will not accommodate this. Alternatively, the Legislature may want to reconsider its 1987 decision to require state agencies to pay sales taxes.
The departments of Finance and Administration have tried to respond to agency complaints and concerns about all of the new computer systems. They have worked on solutions that help agencies get state business done. They have also allowed users to enter certain types of transactions, such as professional/technical contracts, grants, and interagency payments, directly into the new accounting system (GFS) instead of using the procurement system. Finally, the users of all system components reported a relatively high level of satisfaction with the help they received from Mn-ASSIST staff.
Systems Development in State Government
Computer systems development is an evolving field. Unlike constructing a building or a highway, there is a high level of uncertainty and risk associated with such projects. Experts told us that few organizations in either the private or public sector undertake systems development without encountering significant problems. In fact, according to experts at the Gartner Group, a national consulting firm, about 80 percent of all computer development projects "fail" because they are not completed on time, on budget, or in a way that meets user expectations.
The literature on computer systems development suggests that successful projects have the following features: 1) effective executive sponsorship, 2) user involvement and influence, 3) manageable technology and complexity risk, and 4) good project management. In our view, the Statewide Systems Project exhibited some but not all of these characteristics.
Leadership for the Statewide Systems Project was provided by a multi-agency steering committee rather than by a single person or agency. Although the Commissioner of Finance was technically in charge (the Department of Finance received SSP's appropriation), the multi-agency arrangement diffused responsibility and slowed the project's decision-making process. In addition, there was less continuity than desirable in the membership of the steering committee, with at least 10 membership positions turning over during the course of the project. The absence of a single person in charge and the high turnover among the members of the steering committee put the Statewide Systems Project at a higher risk of failure. Nonetheless, the sponsoring agencies believed that no other arrangement would have worked on a project of this magnitude. Other projects we reviewed had one person in charge, although sometimes the person in charge changed several times during the project.
On the other hand, the project succeeded in involving a large number of users in planning and designing the new systems. According to some people we interviewed, this emphasis on user involvement was a reaction to the problems of developing similar large systems where user involvement was low, although others say user involvement was always a key strategy. The Statewide Systems Project also followed other "best practices," such as involving state managers as co-project leaders, using steering committees, designating departmental liaisons, providing for a user review of specifications, and involving users at many points in the design of the system. The results of our survey and interviews suggest that user involvement has been a key factor in ensuring that user satisfaction is at least moderately positive on most measures.
The Statewide Systems Project was highly ambitious. Although there are examples of larger public or private systems development projects, this project was more complex and risky than most. Also, the technology of the SEMA4 component was new and untested in a state government situation like Minnesota's. In addition, the various components of the project were implemented virtually simultaneously, rather than incrementally. One consultant told us that the Statewide Systems Project was simply "unprecedented."
The result of this complexity was a large number of changes to the scope and specifications of the computer systems as the project proceeded. As we have seen, the changes contributed to higher costs than were originally anticipated. Overall, the ambitious and complex nature of the Statewide Systems Project put the whole project at a greater risk of failure.
The size and complexity of the Statewide Systems Project made the project very difficult to manage. In fact, the project was suspended for four months in 1993 because the costs exceeded the appropriation, and the project management decided not to proceed without legislative approval. There was conflict between the state team and the consultant as they constantly negotiated what work was within the scope of the contract and what was not. Also, the decision to modify some of the state's business practices could have been made earlier (as it was in several other large systems projects undertaken by the state). And user training, a critical component of successful systems development and implementation, was criticized by many trainees because the materials for the accounting and procurement training sessions were inconsistent with the way the systems actually worked. SEMA4 training was much better, according to users.
On the other hand, the Statewide Systems Project utilized a range of "best practices" techniques, including a variety of change management strategies to aid in the transition between the old and new systems, an active communications component, and a structured systems development methodology. The project also conducted an internal risk assessment (in addition to the external risk assessment authorized and funded by the Legislature). These steps eased the transition to the new systems and reduced the overall amount of risk involved in the systems development process.
Despite the risks of failure that accompanied the Statewide Systems Project, the project has been virtually completed and is functioning in a moderately successful fashion. However, its size and complexity contributed to its higher than anticipated costs, and one component (the procurement system) needs to be re-examined.
In our view, the state of Minnesota should avoid computer development projects of this scope in the future. Projects that are developed in stages probably offer a greater chance of success, and smaller projects present less uncertainty about costs. We recommend that:
Finally, we think both the executive and legislative branches need to acknowledge this hard reality: computer systems are never permanent. Even after new systems are developed and implemented, they often need adjustments and "fixes," and in time they will need to be replaced. In short, investment in computer systems is an ongoing, rather than a one-time, expenditure, especially for large complex organizations like Minnesota state government.
The Program Evaluation Division was directed to conduct this study by the Legislative Audit Commission on May, 1996. For a copy of the full report, entitled "Statewide Systems Project", (97-05), 81 pp., published on February 6, 1997, please call 651/296-4708, e-mail Legislative.Auditor@state.mn.us, or write to Office of the Legislative Auditor, 658 Cedar St., St. Paul, MN 55155.
Staff who worked on this project were Tom Walstrom (project manager), Jan Sandberg, and Nicholas Franco.