Client
The Mayor's Office for Economic Opportunity (now named NYC Opportunity) was mandated by Mayor Bill Deblasio in May 2017 to use evidence-based principles to combat poverty and increase equality in NYC.
The Civic Service Design Studio is a unit within NYC Opportunity that uses principles in UX and Service Design to make public services more "effective, accessible and simple for all New Yorkers."
Role
User-researcher
Problem
The Service Design Studio at NYC Opportunity deals with unnecessary hurdles and inefficiencies when conducting user-research interviews. I was honored to collaborate with the Studio to see how we could standardize an efficient process for doing user-research.
Solution
We came up with a participatory-designed framework for standardizing the process that the Studio can now use for researching, ideating, and designing solutions in the civic-space.
With a revamped design process, The Studio began to take more actionable notes, incorporate diverse perspectives by working with stakeholders, and test well-informed solution prototypes.
Table of contents
- The Problem
- The Process
- Recruiting participants to interview
- Making sense of the data
- Bringing the team together to design a solution
- Testing the impact of our solution
- The iteration
- v2 and beyond
The problem
Roadblocks throughout the research process
Duplications cause inefficiencies. The Studio's approach to user-research was filled with roadblocks that made collecting and managing user-data tedious and duplicative. For example, a common workflow after conducting a user-interview would be to tag key insights on sticky notes, record them on a cloud-based service (i.e. Google Docs), and then dispose of them in the bin afterwards. That's a lot of wasted post-its.
Too many UX tools to choose from
With so many moving parts throughout the research process, the Studio needed a tool to centralize all quotes, insights, and discoveries in one repository and share it with the rest of the team. However, there were too many software vendors to choose from and we needed to pick the right one that would solve for our specific needs.
Not many actionable takeaways from weekly research-synthesis meetings
Every week, the Studio would meet with key stakeholders on a project to discuss findings gathered during user-interviews. While it was important to have everyone at the table share quotes and insights, the format of the meetings was structured around passive listening and note-taking.
The meetings were in need of a revamped framework that would enable our Studio team and stakeholders to do the following:
- take actionable notes
- ask informed questions
- generate hypothesis
- ideate solutions
The Process- Research design
Recruiting participants for qualitative interviews
I reached out to four of the Studio's designers: two of whom worked full-time and two who were on a short-term fellowship. I liked that there was variation in how long they’ve been at the Studio. In my mind, the recent fellows would most likely have a different take on how things are done as opposed to the more established designers who may have grown accustomed to the same way of doing things. Fresh minds beget fresh perspectives.
What did I want to understand?
In order to understand which UX Tools would best facilitate the Studio’s research process, I had to understand how the Studio did research in the first place. All of my research efforts were focused on understanding three components: People, Processes, and Tools, and how they all affect each other.
I wanted to center my research efforts on understanding my colleagues better: to know what their design process is like, which tools they use, and which possibilities they'd like to explore for solving their pain-points and accomplishing their goals.
The Process- Synthesis
Making sense of the data
After conducting the first round of user interviews, I noticed recurring pain-points that everyone in the team was having, as well as opportunities for solving them.
Here's what the Studio team expressed as their shared pain-points:
“We need more structure on how to formalize our tools and processes. A lot is missing because a lot of people are involved on the research side. How we defined insights were varied.”
"Lack of consistency is what's most difficult about managing data."
"It wasn't consistent. The consistency wasn't there, feeling that the synthesis process was different for the next round"
"It wasn't consistent. The consistency wasn't there, feeling that the synthesis process was different for the next round"
"A lot of data, and really overwhelming"
"Lets say we limited to 5 insights and opportunities, could we have really captured that instead of overwhelming ourselves with 40-50 per person? Could we have done with with 5-10 insights, I don't know. Maybe not. It was definitely overwhelming."
"After the synthesis and capturing that synthesis, we were really exhausted because it took 4-6 hours a day. How do you take that and take a snapshot of it and have it synthesized for you?"
"There's a big amount of raw data in everyone's head that no one else knows about"
"That’s when most of the research was being conducted. Since I’m the project lead, I was leading some of the synthesis sessions, but when I was gone, the team was using different frameworks every time. And they were also conducting research while they were doing synthesis, and it wasn’t paced very well."
"One thing that was challenging for us- it was time-consuming to transcribe into note-taking form and then bringing it into post-its, that whole process was cumbersome"
"I really wanted to organize this, but just couldn't for little technical reasons like your stickies would start falling out if you stacked too many on top of one another. We don't have our own room so we acn't have stuff up for an extended amount of time. So you're constantly moving paper with paper, so stuff starts falling down, you're losing it"
"We had these 6 hour long days, and we were really tired and all this stuff would be up and hanging and we'd be like, 'oh we'll return to it tomorrow', but we'd have 800 other things so it would stay up and wait till the next week that'd we'd block and then we'd forget. We'd have to start over and we'd ask 'what is this again?' "
Here's what the Studio team expressed as opportunities to explore
"Having a designated space for your research- the space was such a restraint. This was the only space that allowed for everyone to use to do synthesis"
"Workflow wise, collaboration is super key- having another at least one or two people in the team working with you so that you can bounce ideas off, having that collaboration so that you're not doing everyting by yourself, even in the synthesis aspect."
"When possible, streamlining your notes and typing directly into a computer and then having time blocked off right after an interview to digitize and then that becomes a norm or if that's something you do during debrief with your partner, which is to type your notes right away during the debrief?"
"Have set synthesis days during a sprint, so you know that every week you have a 4 hour block for synthesis that you know you're coming to the session so you can prepare accordingly"
The Process - co-designing a solution
Bringing the team together to discuss our key findings
After sorting the key-findings into painpoints and opportunities, I wanted to bring the Studio together for a workshop in which we could all offer input to solve for those pain-points and capitalize on opportunities.
Validating our painpoints together along a matrix
Next I presented the team's pain-points and issues. I wanted to see how often they came up in their work and how they felt about them. I asked the following two questions:
- How painful are these issues
- How often do you encounter them
We came together as a team and placed our pain-points on a four-quadrant matrix along the lines of "intensity versus frequency". The vertical axis labelled the two extremes of frequency: “Rarely” and “Always”- and the horizontal one labelled the extremes of intensity: “not a pain” and “ouch, big pain.”
.jpg)
.jpg)
The most pressing and frequent issues were the following:
"Lack of consistent standards, processes, and defined outputs"
"Onboarding new stakeholders and partners in the research process"
"Physical exhaustion when sorting sticky-notes for hours when synthesizing data"
"Rotating teams and positions within teams"
What should we "stop, start, and continue" in our design processes?
The team was then invited to deliberate on how to map out an ideal workflow that would ensure consistency and solve for the above pain-points by asking the following questions:
- What do we want to stop doing?
- What do we want to start doing?
- What do we want to continue doing?
Here's how the team envisioned a standardized process going forward
Before we conduct a user-interview, we should...
- Have a team with diverse work-expertise so that various stakeholders can get involved
- Clarify and align on understanding between interview-lead and note-taker
When we finish conducting an interview we should...
- Surface initial and most revelatory insights and quotes
- Generate fifteen immediate insights on sticky notes
- Take 5 minutes to debrief
- Interviewer sends a "thank you" note and follow-up information to keep stakeholders invested
When we sort through data we should...
- Have a near-transcript record of conversation
- Tag major themes and insights
- Organize info into research themes
When we discuss our findings as a team we should...
- Create discussion-frameworks so that workshops and meetings are actionable, engaging, and not passive.
- Incorporate discussion points on patterns, insights, open questions, and hypotheses/ideas
We ideated the following framework for taking actionable notes in order to be less reliant on passive listening:
Each person in the meeting is given a ‘Notes Template’ to fill out as they are listening to the interviews being read aloud. The template will be divided into the following sections:
Patterns
What data has been validated?
Insights
Which quotes did you find novel/ interesting? why?
Open ?'s
What do you still not know?
Open ?'s
What do you still not know?
Deciding on outcomes and output
One key takeaway from our brainstorm session was that there was a need to track and share data that was emerging from interviews with stakeholders. Together, we decided that we needed to create a repository of emerging data and insights that would be "open-sourced" with key stakeholders and participants.
This gave way to the "Field Logs" which the Studio now uses to journal each important milestone during the design-process for any given project.
The Process- Evaluation our solution
Putting our new design framework to the test- we saw the following improvements:
The Studio was commissioned by the NYC Administration for Childrens' Services (ACS) to work on Pathways to Prevention, an initiative that would allow families to use their "voice and choice" to opt for social-services and thereby prevent their children from entering the foster-care system.
Better, actionable notes called "Field Logs" during qualitative interviews and synthesis meetings
After coming to terms with the fact that we needed to make our share-out meetings more actionable and less dependent on passive-listening, the Studio began to create note templates for all participants in order to:
- ask insightful questions
- identify emerging patterns and insights/trends
- generate hypothesis and ideas
The quality of follow-up questions was higher- helping the team explore solutions that addressed the needs of family stakeholders in a more meaningful way.
Here are some of the questions the team thought of before prototyping solutions:
On Family "voice and choice"
How might we provide parents with opportunities to make informed choices about key parts of their prevention experience?
Trust and safety for families
How do we ensure families safe enough to share their preferences for prevention services?
Personalizing prevention-services for families
I wonder how we might design materials that meet each family’s unique situation?
On what families care about most when opting in for services
what content is important to families when signing up for Prevention services?
The team was better informed when testing out prototypes
By focusing on key emerging-trends gathered during interviews, the team was able to quickly prototype ideas that spoke to the needs of their stakeholders and receive validation from case-workers at ACS.
Diverse groups of teams and stakeholders were incorporated in the research process
We took our findings from our initial brainstorming session and decided to include diverse stakeholders throughout the research process.
For the Pathways to Prevention project, the Studio invited families, ACS case-workers, and members from the Community Based Strategies team at ACS for more enriched perspectives.
Each Thursday in February and March the Pathways team will meet with families, prevention provider staff, and Child Protection Specialists at ACS to test a lot of prototypes designed to answer the project's guiding question
A centralized and digitized tool for tracking, sorting, and mining data
Now with a revamped design process in place, we needed a software tool that could help fulfill our goals and address our existing pain-points.
We narrowed our search to the following tools based on a decision-matrix:
- An infinite canvas
- Ability to create "frames" or sections that can contain any piece of data
- Easy to import photos of sticky-notes and include within your canvas
- Collaborate with others
- Easy to group notes into categories
- Not much in terms of statistical analysis
- Freedom to manipulate trello boards
- Create “sprints”, along with interview notes
- Easy to import photos of sticky-notes and include within your canvas
- Tagging notes as you envision it best
- Free service
- Not much you can do with data analysis
- Not designed to be a UX research tool
- You’ll need to get creative and “hack” it to your benefit
- Useful for data-analysis
- Tagging feature allows for data-mining and clustering into similar categories
- Visualizations to help make sense of repeat tags
- Collaborate with others
- Not easy to import existing notes from a Google Doc or Word Doc
- Learning curve for getting used to their tagging system
- Easy to import existing docs for digitizing
- Easy to tag and create tagged-groups
- Visualizations to help make sense of repeat tags
- Collaborate with others
- We have to manually create categories and insights
The Studio decided to go with RealTimeBoard (Miro) as a UX tool to centralize and digitize all of our findings going forward.
Lessons Learned
Don't underestimate the design process
After conducting the first round of user interviews, I noticed recurring pain-points that everyone in the team was having, as well as opportunities for solving them.
To my excitement, it turned out to be much more than a Google search. After hearing my colleagues express their collective pain-points (i.e. working with limited office space, missing sticky-notes, fatigue from spending days sorting post-its)- I realized there were underlying issues that needed to be solved in a user-centered way.
I took this as an invitation to learn about our Studio, my colleagues' design process, and what a collective vision looked like for standardizing our process in the future.
The lesson for me was to never underestimate the need for thoughtful design in our everyday lives.
Ask "why" to get to the root cause
I learned that before engaging in any design work, it's essential to get to the root cause and not be misled into designing a "band-aid" solution for a problem that's deeper than what initially meets the eye.
Consider the following dialogue:
""We need a software tool to enhance the way we manage data."
"Why"
"Because there's a lot of needless duplication and inefficiencies in the way we gather data."
"Why"
"Because the way we in which we obtain data isn't consistent across the projects that we engage with."
"Why"
"Because our design process isn't really defined."
This method of questioning helped me uncover the Studio's true needs: it wasn't about software tools as much as it was about process-improvement and creating shared, best-practices for the entire team.