Designing A Better Mass Vaccination System

How do you vaccinate as many people as possible, as quickly as possible, during a global pandemic?

Hint: Not the way most of the world decided to.

Our team had a pretty negative experience trying to get a vaccine appointment online during the initial rollout in NYC, as did many people we spoke to who live and work in the city.

As designers, it’s pretty difficult for us to drop a bone when it gets stuck in our craw, so we naturally asked ourselves, “Why is this so awful and can we fix it?” Setting aside the many UI faux pas on the hastily cobbled together scheduling sites, we felt there was a more fundamental flaw with the overall architecture of the whole system.

After doing a little digging, it turned out NYC wasn’t alone. The problem was not only national, almost every nation was planning to do pretty much the same thing!

It’s a bit of a mouthful, but we’re calling what they chose to do a “distributed vaccination scheduling system”. Lucky for us, there is a MUCH better way.

The Not-So-Great Way

First, let’s dive in to the issues and architecture of NYC’s distribution system. Feel free to treat this as an example of what not to do.

 

Distributed Vaccine Scheduler Prototype

 

 

In New York, but seemingly universally, governments chose to implement a distributed, lemonade stand style system to vaccinate their people – ie they’re sending supply to a bunch of independent entities in the hope that they’ll set up shop, put out a sign, and thirsty folks will find them.

These systems put the onus on individuals to get vaccinated on their own, finding and scheduling appointments primarily through web portals. Unfortunately, this system is deeply flawed.

Distributed digital-first vaccination scheduling systems are:

• Unequal – They require internet access, technical savvy, dexterous fingers, disposable time, and a number of other prerequisites that are barriers to large portions of the population.
• Unscalable – A huge number of people became eligible to schedule an appointment in NYC on the same day with tons more flooding the appointment sites only a week later. Demand far outstripped supply in high density metropolitan areas, whereas rural areas had open appointments hundreds of miles away.
• Inequitable – The points above contribute to and exacerbate preexisting inequities resulting from previous systems with inequitable foundations.
• Wasteful – Hundreds of websites throughout the country and thousands across the world, all designed to do the same thing – match people (demand) with vaccines (supply) – were created independently from each other, on different technologies. This was a monumental waste of resources.
• Inaccessible – Many of these sites had numerous WCAG 2.1 issues in addition to more conventional anti-user behavior.
• “Dumb” – When you don’t know where demand is, you lose all sorts of consequential logistical and sentiment data. In marketing, we have myriad tools at our disposal to infer sentiment, desires, and intent, but nothing beats someone simply saying, “I want _____”. All ya have to do is ask.
• Invasive – Quite a bit of private, personally identifiable information (PII) was required when scheduling an appointment for a vaccine through the portals we reviewed, regardless of whether it was hosted by a private or public entity. In some cases, users were required to make a binary gender choice in order to access the vaccine (do better Walgreens). And for what? Anyone could have simply lied and considering the lengths people went to get a jab before it was their turn, we can reasonably infer that at least some of that data is garbage. More care should be taken when asking people to hand over their PII. Ask for only the most essential information for any given task instead of calling for as much as possible. Especially while commercial and government entities alike are having their(our) data exfiltrated en masse.

Despite this system, vast quantities of people were inoculated in large quantities, but we believe credit is owed more to the people on the ground than the system’s design. That said, at some point in a campaign, or in some locations, this system could potentially work better than ours. However, in high density communities, it was objectively a bad solution to lead with.

This system is based on 3 erroneous assumptions:

1. 1. People are busy and will want to schedule a time that’s convenient for them.
   2. Enough supply and resources would be available to meet demand, uninterrupted.
   3. Demand is distributed evenly geographically.

We know now that (some) people simply wanted to get vaccinated as soon as possible, demand was asymmetric geographically, and supply/resources were insufficient to meet demand at all times. All of these were knowable in advance and should have been accounted for in a purpose built distribution system.

Architecture

Our team tends to view system architecture through the lens of our primary audience, which is why our architecture diagrams are called “User Flows”. A user flow helps visualize the important steps a user would experience in any given system and reveals the architecture of that system. The distributed system’s user flows below have been reengineered based on what we experienced in NY and can, to some degree, be thought of as a kind of site map for the prototype above. There are two levels of simplification, but they are essentially the same flow. The extra simplified diagram was largely created to facilitate comparison with our system.

We recommend viewing the Simplified and Extra Simplified flows in our presentation format on Figma.

 

View Distributed Scheduler User Flows

 

Extra Simplified

 

When we design a system, we tend to focus on the user’s needs while seeking a balance with an organization’s goals and capabilities. We also tend to investigate and solicit feedback often along the way. This allows us to identify mistakes, errors in assumption, and any other issues quickly, early, and throughout the design and development process.

It’s pretty clear to us that none of that was done here. Especially at the federal level.

The federal level’s vaccine finder is a case in point. It may look nice, but any user visiting that site would have expected it to show them where a vaccine was currently available in real time. In reality, the data was 3 days old and completely useless. Even if it was miraculously correct, the user would have to actually make it to one of the distributor sites in time to reserve their spot. If they failed, they’d have to start all over, most likely having to learn a different platform’s unique user flow from scratch. Even if the new availability was on the same site, the user still had to reinsert all their information during every attempt, even if it was in the course of a single session. A poor experience we suspect is the reason for the W3C adding redundant entry to the WCAG2.2 draft in May of this year.

To summarize, this system was never designed to quickly vaccinate people at scale. It was designed to pass the buck down the line, letting state and local institutions figure it out on their own. Unfortunately, most were not equipped to do so and the result was a system that put the brunt of responsibility on individuals.

Booster Shots

With booster shots on the horizon and few if any changes to the systems we’ve reviewed, we’re all most likely going to have to run this unnecessary gauntlet again with even more complications than before considering none of these portals were primarily designed to accommodate boosters in the first place. It begs many questions that will now need to be answered and dug up by everyone who needs a boost.

Do you go to the same place as before? What if they don’t have supply? What if it’s no longer operating? What if they only have a different type? Will they reach out to us or will we have to find an appointment like we did before? Will we have to provide all of our PII…again?? What if you moved?

Apparently we need actual systems designers in the body that governs the systems we live by.

A Better Way

Let’s walk through how we developed our system to better understand why it is the better way.

Goals

In user experience design, we identify the user and organization’s primary objectives/goals, categorize them hierarchically, and make sure that the primary objectives are clearly defined.

In this case, the primary user and client objectives couldn’t be more clear. (For the most part. We’ll get to the anti-vaxxers and vaccine-hesitant later on.)

User

• Get vaccinated as soon as possible
  • We also assume
    • Most convenient (nearest location, when the user is able)

Client

• Vaccinate as many people as possible as quickly as possible
  • We also assume
    • Protect people, save lives
    • Efficiently

Happily, everyone’s goals align! 

Well, not quite. In reality, every user will have different primary objectives at different times. Some people will want to learn more about the vaccines, side effects, the current state of vaccination in their area, whether what they heard Aunt Sally say is true, or to simply manage an appointment they already made. Hence a hierarchical system. Since all of these goals can’t be the most important, ie the first thing someone sees, we pick one of them to be on top, and address the rest in a logical, familiar order. 

In addition to the user’s additional goals, the client will also have a number of additional requirements and objectives to consider when developing a system. For example, they might require that the system be ethical, equitable, adaptable, efficient, and executable in a timely manner.  

Once we have these objectives, we can come up with more effective solutions to help our user and client reach their goals as efficiently as possible.

Assumptions

Designers also have to imagine what an experience will be like for a user. This requires us to make certain assumptions about the folks we’re designing for and the environment it’ll be experienced in. 

Our Assumptions

• Initial Constants:
  • Low Supply
  • High Demand
  • US specific: user healthcare is relatively self directed
• User wants/needs:
  • ASAP
  • Nearest (most convenient) location
  • Knows when they are typically available or can make themselves available
  • Transportation Method/Ability
  • Contact Method/Ability
  • Potentially, vaccine preference
• Client wants/needs:
  • Ethical & Equitable system
  • Supply/Demand flexibility
  • Practical and efficient distribution system
  • Live awareness of demand/uptake/vaccination status/virus spread/etc
    • Essentially a comprehensive BI system

We then build user flows and prototypes of our ideas and ask real people to test them in order to validate those assumptions and/or refine them until we have something that works. It’s a process that’s fairly similar to the scientific method’s iterative approach.

Architecture

Here’s where we start putting it all together. Mixing in what we know about our user and client’s goals with our assumptions, then sprinkling modern advertising, design, and behavioral theory on top.

The latter is required to get our user to take action (get vaccinated). Without it, all effort falls in a forest without ears. 

To simplify the industry bit, we need to move a user from Awareness (“OMG there’s a deadly virus that can kill me and they [client] have a vaccine.”) to Action (“Yay! I’m safe!”).

Crucially, and somewhat implicitly, the designer of this system needs to create a flow from Awareness to Action that is as frictionless as possible for a user.

Friction is anathema in design. Friction leads to frustration, frustration leads to suffering, suffering leads to resentment, and resentment can even lead to hatred.

Obviously a little hyperbolic, but you get the point. A designer’s job is to respectfully solve problems, not create new ones.

The solution that most effectively satisfies the primary objectives for both parties with the above assumptions and the least amount of friction is, and always was, a Centralized Pre-registration Scheduler.

What is a “Centralized Pre-registration Scheduler”?

In short, you ask people to go to your website/app, fill in a little info about themselves, what times they’re available, and then tell them where to go when it’s their turn.

That’s it.

Crucially, you need to collect the following from each user:

• Contact Info
• Location Data
  • Preferred Site (if you already know where your Vax sites will be)
  • Otherwise, user’s city, zipcode, etc
• Availability
• Transportation Method (theoretically optional, but for maximum equity, pretty important. Someone in a car could go 30 miles no problem, but public transport, bikes, assisted, walking, can’t move…not so much.)
• Personal Info (the least amount necessary for segmenting based on risk priority tiers)

And it looks like this:

We recommend viewing the Simplified and Detailed flows in our presentation format on Figma.

 

View Centralized Scheduler User Flows

 

Simplified

 

Benefits of this system:

As you can see, this system is significantly simpler than a distributed system from a user’s perspective.

Users can now sign up at their leisure before vaccines are even available, avoiding the chaos that ensues when everyone digitally stampedes for the earliest available appointment. Aside from the minor inconvenience of the initial information sharing/account setup, there is no further action required by a user other than confirming the appointment given to them and showing up. Short of sending a nurse to everyone’s house (obviously not possible), this UX provides the least possible friction for a user, while greatly enhancing the capabilities of the client.

It also has the benefit of being implementable at a national level, reducing the wasted time and resources poured into every site created by state, local, and corporate organizations while still benefiting from their physical infrastructure.

All else being equal, for each priority tier, a preregistration system is significantly more equitable as well.

Instead of the fastest computer or the newest sneaker bot snapping up the earliest appointment, everyone who has preregistered gets assigned an appointment randomly. When fighting bots, preregistration and random assignment provide extra time to identify illegitimate accounts and essentially eliminate the incentive to use a bot, since being first provides no additional, guaranteed benefit. It also unlocks collecting preregistrations through non-digital means, like phone and physical preregistration sites! This breaks down the inequitable access afforded to the digital “haves” that the primarily digital-first, distributed distribution system relied on.

Since location data is being collected simultaneously with demand, there’s the added bonus of knowing when & where to send supply as well, dramatically improving logistics intelligence. Especially useful when supply is severely limited relative to demand, as was the case during early stages of the US rollout and, frankly, still an issue globally.

Location based demand provides data for identifying the best placements for max vaccination sites, allocating the appropriate # of people to those areas, and supplying them with the training and resources they need. When combined with infection models during a surge in cases, distribution and resources can be dynamically directed toward areas with higher risk profiles as well, potentially decreasing the rate of severe cases at the precise time and place it’s needed most.

Location based demand also acts as a kind of sentiment early warning system. Identifying areas at risk for hesitancy early lets you allocate more resources and advocacy to those communities, before it becomes more problematic. It also tells you where you may have awareness problems, allowing you to efficiently allocate ad spend and additional marketing efforts. The real benefit to outreach comes from the early combination of location based preregistration velocity with location based marketing efforts. Knowing what works, early and in relatively real time, makes for a more nimble campaign that’s better able to address issues and changing information as quickly as possible.

Despite the many benefits of this system, we’re human and we’ll inevitably mess up.

When we need boosters or some other virus/variant tries to dethrone us, this system could theoretically be used to ping everyone that signed up last time directly. Awareness is still going to be necessary, but the vast majority of people who registered are going to have the same exact contact info they provided the first time around, giving the campaign a huge head start. That said, user’s should be allowed to delete any stored data or have it automatically erased after a set period of time.

Better yet, don’t store any user data at all:

You can make an app used by many millions of people that has no data…Cool chart by @forbes & @UKZak 🙈🙊🙉 https://t.co/gWFqyIeoZ3 pic.twitter.com/Unngddaq5M

— Signal (@signalapp) January 5, 2021

With all the hacking in the last couple years, and warranted lack of trust in government entities, encrypted on device data storage might even be a requirement for wide adoption of a system like this.

How Do You Know It’s Better?

Well, it turns out, we weren’t the only ones to think this up. While we independently came up with this system using a first principles approach, we were made aware of Canada’s.

Turns out it is very similar to the core aspects of our system!

Canada may have gotten off to a slow start, but they currently have one of the highest vaccination rates on the planet. Especially when factoring in their total population size (~38 million). As of writing, of eligible Canadians 12 and up, 84% had received 1 dose and 77% were fully vaccinated. This drops to 73% and 67% respectively when counting the full population, which places them just a hair outside of the current top 10.

Yes, Canada has national healthcare and the US doesn’t, but there’s really no reason why a centralized preregistration system wouldn’t have worked just as well in the US.

It’s possible lower vaccine hesitancy in Canada’s general population also contributed to their higher rates, but there’s another potential factor at play from the world of behavioral economics: Loss Aversion. In general, humans don’t like it when their belongings are taken from them. They also tend to value keeping something they already have, more than gaining something they didn’t have before. Since eligible Canadians were basically told, “Hey, your vaccine’s ready. Show up [here] at [this time] or ya lose it.” an inherent bias may have been triggered that resulted in increased uptake and the high vaccination rate we see today.

Of course, without a proper study we may never know.

With the right setup, a version of a centralized preregistration system could have worked globally as well, assuming most nations cooperated with each other and user data wasn’t crossing borders.

To be critical of Canada for a moment before wrapping up. Eligible Indigenous people living on reserves are currently lagging behind full vaccination rates by around 5% compared to the national average, despite prioritization during early parts of the campaign. And what about vaccination rates for the off-reserve population? That’s ~78% of the total Indigenous population in Canada, but it’s not being documented at the national level.

Good intentions need to be more than words and government planners globally need to concentrate on follow-through.

Conclusion

Based on our experience with various procurement systems, it’s pretty likely that every governing body and corporate entity getting their hands on supply, had someone write up an RFP about something they had no idea how to build. Writing RFP’s looking for vendors that specialized in physical logistics makes obvious sense, especially for such a delicate vaccine with pretty ridiculous temperature requirements like the mRNAs. However, making that vendor responsible for developing the overall architecture and UX for a digital system might not have been the best move. Traditional logistics expertise does not equal digital logistics expertise.

Accommodating old school logistics vendors needs over users and poorly scoped RFPs are most likely what led to the scourge of distributed vaccination appointment schedulers. The odds that they specifically asked someone to create a vaccination scheduling website instead of “the most efficient way to vaccinate as many people as possible” are pretty high considering what we got. 

Next time you have to build a digital product for billions of people, maybe call up Facebook, Google, Apple, Amazon, AirBNB, (or literally any experienced UX designer worth their salt…hi) etc and ask them for help (but obviously don’t let them have access to any user data). Through no fault of their own, elected officials clearly aren’t qualified to design and test a system of this scale themselves. 

Maybe also don’t ignore a group that distributes billions of vaccines every year when they say treating vaccine distribution in an uncoordinated fashion is bad for everyone.

Even the HHS distribution plan, “From the factory to the front lines” was centralized. Why then was the digital portion left up to distributees*?

Next time, just make the whole system centralized.

 

*To be fair, HHS did have something resembling a centralized digital plan. The CDC was supposed to facilitate distribution through their “finder” web portal, but as we noted in our distributed user flows, it was woefully inadequate. With useless, 3 day old supply information, it was barely a phone book, let alone a vaccine “finder”. A centralized digital distribution system it was not. As of posting, the latest version of the “VaccineFinder” was moved to vaccines.gov, rebranded and redesigned, and is still just as simple as it was before. It still shows 3 day old information and shuttles users off to whichever leg of the chain they happen to click on to see if they really do have supply.