Ticking along with ESD

by Bruce Miller (owner of this web site)

What makes ESD tick or not?

Here’s some insight based on publicly accessible documents.

Washington State’s primary data center is at 1500 Jefferson St, SE, Olympia, WA. “RCW 43.105.375 requires agencies to migrate all existing servers into the SDC [State Data Center] (as well as locate all new servers within the SDC).” This data center was completed around 2015. ESD moved into this new data center around 2017. Source

Before the new data center ESD was using a mainframe computer in another building. A report in 2014 described the situation: “The Employment Security Department has an aging mainframe benefits management platform that is costly, difficult to enhance and maintain, and increasingly unsuited to supporting the needs of a modern Unemployment Insurance (UI) program. Web-based claims applications have been added around the periphery of this pre-Internet system. . . . The legacy mainframe Unemployment Insurance (UI) tax system and related ancillary systems are being replaced by a modernized Next Generation Tax System (NGTS). This system is built on a modern Service Oriented Architecture utilizing Microsoft .Net platform. NGTS will account for and collect UI taxes within a framework that will provide consistent application of business rules and flexibility to modify those rules to accommodate changes in the law, regulation, or business practice.” Source

In 2014 ESD launched the Next Generation Tax System (NGTS), used primarily by business.Source

Problems? Yes. Programming errors were doing weird things. . “For example, NGTS considers any empty field on an unemployment tax return to be an error… which sounds okay except the standard paper form provides six spots for six employees. So you have to fill in all six spots if you file your return using paper. You can’t avoid the penalty if, oh, say you have only one or two employees.” See: Washington State Employment Security Department: Computer System Meltdown?

Two years later — still problems. One business was due a $144.59 refund. Took nearly a year to get it because the system didn’t know how to issue a refund. See: Years late and millions over budget, state computer system called a ‘failure'” In the KING 5 story: “NGTS was supposed to take three years to complete, with a budget of $46.8 million. Instead, it’s now been eight years and NGTS has is more than $20 million over the original budget.”

In April 2015 ESD bought “FastUI, a commercial off the shelf system (COTS) procured from FAST Enterprises, LLC [in Colorado] in April 2015.” FASTUI would replace the old GUIDE system. Source

Among other functions, FASTUI would become the primary system ESD claimants interact with for applying for and managing claims. FASTUI is a component of a larger ESD technology project called UTAB: Unemployment Tax and Benefits.

FASTUI was launched in January 2017. It had problems to, reported on by KING 5 in this article: Thousands can’t access unemployment benefits due to system failures.

KIRO also had a story with other details. “While 37,000 people were able to file claims over the phone, viewers told KIRO 7 the newly upgraded website is allowing users to log on but then they are unable to get passed the identity verification step, so they get locked out.” KIRO also said “The state says the call center received nearly 750,000 phone calls on Tuesday. The Employment Security Department says about 60,000 people receive unemployment benefits and were calling multiple times.”

The identity system mentioned in the KIRO story is Secure Access Washington, known in shorthand as SAW. Because ESD claimants must have a SAW account before doing anything in the ESD system, most claimants figure SAW is a part of ESD. According to one document, SAW is the responsiblity of Washington Technology Solutions, not ESD. “As ESD is reliant on WaTech for services such as Secure Access Washington (SAW) . . . .” Source

During all this time into January 2017, ESD never had a disaster recovery plan. This falls under the broad concept of business continuity. (See: “Computer Ignites: Panic” below explaning the difference between backup and continuity — from this author’s own disaster.) How can an organization keep running in times of disaster, technical failures — all sorts of conditions. Recognizing this a funding proposal was created in 2018 for a disaster recovery system. Among the salient points in the proposal:

* “ESD does not currently have a DR/COOP (Disaster Recovery/Continuity of Operations) solution in place . . . . ”

* Complicating DR/COOP are system constraints: “Due to design and functionality constraints, on premise business applications such as NGTS and UTAB are currently not capable of leveraging cloud services such as SaaS [Software as a Service] or IaaS [Infrastructure as a Service] for production [meaning live if need be] DR/COOP. Redundancy solutions for these applications incorporate traditional data center design architecture similar to what is currently deployed in the SDC.” Source

In other words, scaling up in the cloud is not possible for all components of UTAB.

The ESD plan was to basically replicate in the Quincy Data Center (QDC) the operations taking place in the State Data Center in Olympia (SDC).

“Initial focus will be on QDC infrastructure, which will be used to replicate current core network services located within the SDC. QDC will be architected in a ‘hot’ configuration; meaning all associated infrastructure will be leveraged to augment ESD core IT enterprise services used in daily operations. In the event of an emergency or disruption, core IT enterprise services will be configured to fail over to the backup location automatically, resulting in minimal interruption in services to customers. Cloud services will also be leveraged, however extensive research and development in cloud services will be required in order to determine the most robust solution possible to maintain integrity, sustainability and operability of IT assets, applications and business services unique to customer requirements of ESD.” Source

The plan was funded and began in the fall of 2019 and to be complete in 2021. Creating the DR/COOP in the QDC would be complex, as suggested by a February 20, 2020 ESD consultant report, explaining that ESD “has over 170 applications/systems providing both direct support to Washington residents as well as administrative support to the operation of the Department, including employee payroll and vendor payments. Customer-facing services include unemployment insurance benefits, paid family medical leave benefits, job placement, training, vocational rehabilitation, and labor market information.” Source

As late as March 25, 2020, reported by the same consultant, ESD was still in the same state of affairs regarding DR/COOP:

Currently ESD does not have a back-up or disaster recovery (DR) plan that would enable ESD to pay unemployment benefits to Washington residents in a timely manner in the event IT infrastructure, systems, or data became unavailable. [Emphasis added.] The current back-up solution includes snapshots of data every 24-hours stored in one of ESD’s claim centers. Recovering the snapshots would require a manual process taking weeks to months to return to an operational state. Interruption to or total loss of technology systems and associated IT infrastructure would significantly disrupt critical services to job seekers and employers, potentially causing significant social and economic impact to residents of Washington State”. Source


Computer Ignites: Panic

In the afternoon of August 21, 2016 smoke alarms went off in the house. I opened the door to my office. The room was filling with black smoke. On the far side of the room was my main computer — a mid-sized tower weighing 42 pounds — spitting flame out the front through an open drive bay. After a fumbled attempt with an fire extinguisher, I picked up the heavy high-back office chair and handed to my housemate to get it out of the way. I began to disconnect all cables into the computer so I could take it outside. The front door was open to help clear the smoke, so with the chair gone, I would be able to do a fast exit. I picked up the computer still flaming and walked it out into the front yard.

Within hours — mostly after settling my nerves — I sat down at another work station in the house and began work. I lost no data. And my data was synchronized on other equipment and systems. The key to business continuity is data that is not only backed up, but synchronized on near identical systems. What ESD lacks is synchronization — a goal in the DR/COOP project.

More about me here.


Author’s thoughts

Adding to the complexity of numerous systems during the COVID pandemic is the need to implement social-distancing among workers. Many ESD claims workers we are told have moved to working from home. This could create all sorts of problems. Brainstorming:

Are workers required to use an ESD issued computer (most likely a laptop). If yes, then each one needs to be configured. If yes, each one migh require a VPN (virtual private network) connection. More setup on laptops and potentially at the incoming connection into the data center.

Are home-workers taking phone calls? This is another potential logistical nightmare. How will the system keep people on hold and then route the call to vacant phone? Will the phone system allow for outgoing calls? If the inbound calls are simply forwarded to a worker’s personal phone, then there is probably no outbound calling ability. No worker would want their personal number showing up in caller ID.

Do the connections from home allow connections to ALL the systems a worker would normally use? Perhaps a system for sending messages through eServices is not possible with the home setup?


Other ESD info I’ve put up:

Enhancing Seattle Times ESD Article on Missed Red Flags

WA State CIO apparently OK with ID sent by email

Washington State Sets Unemployment Benefit to $0 by making up job title

WA State ESD Commissioner on Adjudication

Washington State Sets Unemployment Benefit to $0 by making up job title

Obvious programming errors in PUA application.