ESD Notes

Phones
Personal thought on the phone situation
A large percentage of the case workers are working from home. Very possible the phone system handling incoming calls for voice is a logistical nightmare and reliability connecting to the case workers is not stable, unlike if in an office situation where every part of the phone system can be standardized and on dependable connections.

Past articles

2014-10-16: Washington State Employment Security Department: Computer System Meltdown?

2016-11-02: Years late and millions over budget, state computer system called a ‘failure’

2017-01-04: Thousands can’t access unemployment benefits due to system failures

2017-01-05: State unemployment computer system still not working

Excerpts from various documents

Search for part of the excerpt to find in the document.

UTAB Source

IT COOP Source

UTAB: Unemployment Tax and Benefits Project MONTHLY QUALITY ASSURANCE REPORT DECEMBER 1, 2016 TO JANUARY 3, 2017

January 17, 2017

There is still concern regarding whether the [SAW] system will be able to manage a large number of concurrent users.

ITCOOP: Decision Package Bundle

Sep 15, 2018

Background:
All computer, storage, network and security hardware supporting Employment Security Department (ESD) business operations are currently housed within the State Data Center (SDC) in Olympia, WA. The SDC supports over 700 servers, 900TB of storage, telephony, critical data networking and security equipment providing support for crucial applications and business services such as Next Generation Tax System (NGTS), Unemployment Tax and Benefits (UTAB), and Paid Family and Medical Leave (PFML). Additionally, all of ESD campus and remote sites receive SGN and internet access via the WaTech provided Wide Area Network (WAN), which logically connect through the SDC. In the event of a catastrophic event impacting the SDC, virtually all ESD business operations and customer service delivery would be negatively impacted, resulting in significant degradation of unemployment, WorkSource, and family and medical leave services to employers and jobseekers

Current State:
In the most recent legislative session, $82 million was appropriated for ESD to implement a PFML application, which will stage Washington to become only the fifth state in the nation to adopt a sanctioned paid family leave program. During fiscal year 2017, ESD core applications combined processed about $1 billion in unemployment insurance premium payments. Legislative funding, including the way ESD provides critical serviceability may be refined and developed in the continuum and execution of core services, seamless and transparent to Washington constituents.

Currently there is no functional DR solution in place supporting ESD technology services or the underlying IT infrastructure. The current approved COOP plan for ESD enterprise data is to replicate from the SDC across the WAN to a legacy storage array located in the Spokane Claim Center. In the event of a disaster or loss of data within the SDC, ESD enterprise systems and associated business services could only be restored after the application of a significant amount of monetary and personnel resources. During this restoral time, ESD would not be able to provide essential business services to the employers of job seekers of Washington State.

Proposed solution:
This decision package request seeks to leverage aspects of Quincy Data Center (QDC) and cloud services to enhance survivability, mean time to recovery, agility and return on investment for systems and hardware lifecycle. Initial focus will be on QDC infrastructure, which will be used to replicate current core network services located within the SDC. QDC will be architected in a ‘hot’ configuration; meaning all associated infrastructure will be leveraged to augment ESD core IT enterprise services used in daily operations. In the event of an emergency or disruption, core IT enterprise services will be configured to fail over to the backup location automatically, resulting in minimal interruption in services to customers. Cloud services will also be leveraged, however extensive research and development in cloud services will be required in order to determine the most robust solution possible to maintain integrity, sustainability and operability of IT assets, applications and business services unique to customer requirements of ESD.

Additionally, this effort will be implemented in a phased approach in order to fully define requirements, research/validate possible technology solutions, develop architectural plans, coordinate with WaTech and vendors, and most importantly, maintain operational stability for ESD business operations. Initial funding in the first three years focuses on infrastructure equipment and services required to populate the fixed site location. Licensing, software maintenance and support funding is required to support the infrastructure hardware, and averages around one quarter of the equipment costs.Average replacement cycle for hardware is 3 to 5 years.

If the proposal is an expansion or alteration of a current program or service, provide information on the resources now devoted to the program or service.

ESD does not currently have a DR/COOP (Disaster Recovery/Continuity of Operations) solution in place, so no existing resources are devoted.

Please provide a detailed discussion of connections/impacts identified above.
Regional/County Impacts: A robust DR/COOP solution for ESD IT systems establishes redundancy and resilience to the enterprise infrastructure. This ensures the ability for ESD to maintain functionality of critical business applications such as NGTS, UTAB and PFML, and core business services such as UI customer service providing unemployment and tax services to the citizens of Washington State. Additionally, WorkSource offices will remain connected with essential network and telephony services in the event of a failure at the SDC.

Responds to specific task force, report, mandate or exec order:
State of Washington Governor Directive 12-20 and 13-02 direct agencies to ensure performance of the organizational critical functions, including technology systems that support those functions, in the event of a disruption caused by an emergency or disaster.

What alternatives were explored by the agency and why was this option chosen?
Due to design and functionality constraints, on premise business applications such as NGTS and UTAB are currently not capable of leveraging cloud services such as SaaS or IaaS for production DR/COOP. Redundancy solutions for these applications incorporate traditional data center design architecture similar to what is currently deployed in the SDC. However, long term IT vision for the agency incorporates more widespread use of cloud services in order to reduce costs, resource requirements and associated overhead. Validation and implementation of cloud services will result in the ability to add more cost effective and flexible solutions in the agency’s IT portfolio. This proposal defines a path forward using both traditional infrastructure and cloud services, providing crucial redundancy and survivability for IT services.

UTAB: 2017-19 Biennium Budget Decision Package

The former benefits payment system (GUIDE) is being replaced by UTAB, using an “agile” approach. In agile, only basic functionality is deployed in the initial roll-out, so that users receive more immediate benefits for technology investment dollars. Other requirements and features of the system are deferred to subsequent, ongoing implementation. Initial implementation of the UTAB project is scheduled in November 2016. In April 2017, the base system will transition to ongoing maintenance and support.

Agency Recommendation Summary Text:
The Employment Security Department (ESD) requests additional appropriation authority for further agile development of the Unemployment Tax and Benefits System (UTAB). UTAB is the state’s IT system for paying unemployment insurance (UI) benefits to workers who lose their job through no fault of their own. ESD paid $1.0B to over 220,000 workers last year.
The former benefits payment system (GUIDE) is being replaced by UTAB, using an “agile” approach. In agile, only basic functionality is deployed in the initial roll-out, so that users receive more immediate benefits for technology investment dollars. Other requirements and features of the system are deferred to subsequent, ongoing implementation. Initial implementation of the UTAB project is scheduled in November 2016. In April 2017, the base system will transition to ongoing maintenance and support.

Package Description
ESD is replacing its legacy unemployment benefits systems (GUIDE) with UTAB, using FastUI, a commercial off the shelf system (COTS) procured from FAST Enterprises, LLC in April 2015. UTAB is scheduled for initial launch in November 2016, followed by a six-month warranty period. At the conclusion of initial implementation in April 2017, the base system will transition to ongoing maintenance and support.

Decision Package expenditure, FTE and revenue assumptions, calculations and details: Agencies must clearly articulate the workload or policy assumptions used in calculating expenditure and revenue changes proposed.
• Two new dedicated FTE (MA4 level) and two new FTE (ITS5) positions to support additional UTAB system development, maintenance and growth. These positions will meet with the various agencies and stakeholders to determine joint business needs, gather business and technical requirements, manage stakeholders, develop project timelines, and execute communications plans, as well as manage contract amendments and associated deliverables with the FAST team.
• $3.0M for system enhancement development costs through FAST Enterprises, LLC. (system contractor).

What alternatives were explored by the agency and why was this option chosen?
UTAB was built as a commercial off-the-shelf (COTS) system using agile methodology, after extensive consultation with the state OCIO, as the most efficient and effective means of rapidly delivering value to employers and job seekers. Further system enhancements and development performed with ESD ITBI staff and our current FAST Enterprises contractors is the natural progression of this plan.

Agency Recommendation Summary Text:
The Employment Security Department (ESD) requests additional appropriation authority for further agile development of the Unemployment Tax and Benefits System (UTAB). UTAB is the state’s IT system for paying unemployment insurance (UI) benefits to workers who lose their job through no fault of their own. ESD paid $1.0B to over 220,000 workers last year.
The former benefits payment system (GUIDE) is being replaced by UTAB, using an “agile” approach. In agile, only basic functionality is deployed in the initial roll-out, so that users receive more immediate benefits for technology investment dollars. Other requirements and features of the system are deferred to subsequent, ongoing implementation. Initial implementation of the UTAB project is scheduled in November 2016. In April 2017, the base system will transition to ongoing maintenance and support.

UTAB: UTAB Enhancement Project (Unemployment Tax and Benefits)

Monthly QA Report #5
March 2019
April 4, 2019

The DOL-ESD System Architecture review was required by the OCIO and was approved on March 13, 2019. The FAST Enterprises Contract was already in existence when CAI came onto the project.

UTAB: UTAB Enhancement Project (Unemployment Tax and Benefits) Monthly QA Report #2

December 2018
January 16, 2019

The Fast team has brought on more developers and are fully staffed at this time.

UTAB: UTAB Enhancement Project (Unemployment Tax and Benefits) Monthly QA Report #4

February 2019
March 6, 2019

As for the project vendor, the Fast team is fully staffed at this time.

Fast was on the original project in 2017 and is now the vendor for the UTAB project. It is an assumption that to have Fast be the vendor on this project, they must have met all the controls required by ESD. CAI has not had the opportunity to adequately assess this QPA.

UTAB: UTAB Enhancement Project (Unemployment Tax and Benefits) Monthly QA Report #6

Monthly QA Report #6
April 2019
April 30, 2019

FAST software is already part of the ESD Software Portfolio.

ITCOOP: IT CONTINUITY OF OPERATIONS PROJECT PROJECT MANAGEMENT PLAN

February 15, 2020

As ESD is reliant on WaTech for services such as Secure Access Washington (SAW)

As ESD is required by law to store equipment within a state data center, ESD is reliant on WaTech will provide enough enclosures for ESD to house our disaster recovery solution at QDC.

Project Objective
By July 1, 2021 ESD will have the solution, plans, and processes in place and staff prepared to react if a natural disaster occurs, so there’s minimal disruption to service delivery.

ITCOOP: IT Continuity of Operations (IT Coop) Project Quality Assurance Readiness Assessment Report February 2020

February 20, 2020

The Employment Security Department (ESD) has over 170 applications/systems providing both direct support to Washington residents as well as administrative support to the operation of the Department, including employee payroll and vendor payments. Customer-facing services include unemployment insurance benefits, paid family medical leave benefits, job placement, training, vocational rehabilitation, and labor market information.

Currently ESD does not have a back-up or disaster recovery (DR) plan that would enable ESD to pay unemployment benefits to Washington residents in a timely manner in the event IT infrastructure, systems, or data became unavailable. The current back-up solution includes snapshots of data every 24-hours stored in one of ESD’s claim centers. Recovering the snapshots would require a manual process taking weeks to months to return to an operational state. Interruption to or total loss of technology systems and associated IT infrastructure would significantly disrupt critical services to job seekers and employers, potentially causing significant social and economic impact to residents of Washington State.

ITCOOP: Information Technology Continuity of Operations (IT COOP) Project Quality Assurance Readiness Assessment Report January – February 2020

March 25, 2020

The Employment Security Department (ESD) has over 170 applications/systems providing both direct support to Washington residents as well as administrative support to the operation of the Department, including employee payroll and vendor payments. Customer-facing services include unemployment insurance (UI) benefits, paid family medical leave (PFML) benefits, job placement, training, vocational rehabilitation, and labor market information.

Currently ESD does not have a back-up or disaster recovery (DR) plan that would enable ESD to pay unemployment benefits to Washington residents in a timely manner in the event IT infrastructure, systems, or data became unavailable. The current back-up solution includes snapshots of data every 24-hours stored in one of ESD’s claim centers. Recovering the snapshots would require a manual process taking weeks to months to return to an operational state. Interruption to or total loss of technology systems and associated IT infrastructure would significantly disrupt critical services to job seekers and employers, potentially causing significant social and economic impact to residents of Washington State


Bruce’s Primary Home Work Station
Six monitors on one computer. 40″ monitor at top on different computer.