From Feedback to Finish: How to Leverage User Focus Groups for Website Design

find out all right welcome everybody

welcome back from lunch we are here just

so you're sure you're in the right spot

we're going to talk about from feedback

to finish how to leverage using photo

scripts for website design so let's get

into it so really I guess what I wanted

to emphasize other than just reading

through this agenda is that I really

want this to be its interactive as you

folks want it to be this is really a

very personal slash anecdotal but I

think useful for other people too but I

do want to say that this is a very

specific use case study so if you find

that it's not useful for you like 10

minutes into this feel free to go

perception and still be fine but if

you've got questions as we're going

through I did reserve some time at the

end for Q&A please feel free to pop up

with questions in the meantime as I said

this is a pretty specific use case but I

do think that there are some lessons

learned that can be abstracted to other

perhaps smaller efforts as well I also

wanted to mention if you folks haven't

noticed there's also a UX session

tomorrow a camera if it's in the

afternoon or not which might be

complementary to this I'm talking about

smaller scale user focus groups and UX

design experiences so a little bit of

background on me my name is Kelly

Chatterton I'm director of web

experience at Cleary

Partners a little bit about me and a

little bit about clarity I've been in

this space for over 20 years at this

point

I remember when mosaic came out that

people really exciting very very cool I

worked with lots and lots of different

platforms I've done some custom

development but mostly working with

different content management systems and

I've also worn lots and lots of

different

that's so I've been a developer although

honestly at this point that was a long

time ago I've been in QA I've done

helpdesk I've been the pointing your

manager for a while so I'm telling you

this because I have seen heard like the

lots of clients clarity partners has

lots and lots of clients but I did want

to emphasize that Clary partners

primarily works with government public

sector clients and if you haven't worked

with those before they do have their own

set of challenges

they tend to move at a slightly

different pace and say private sector my

private sector clients always want

everything done yesterday or last week

if at all possible nobody comes in the

private sector nobody comes to you and

says you know what thinking about maybe

making a website and maybe in a year or

two we'll do it we just want to talk

about it right now

that's totally what the public sector

clients want to do they want to spend a

lot of time for new things out before

him again I'm telling you this because

the rhythm of this particular case study

project that we're going to talk about

might not be the rhythm of the projects

that you're working with on a day to day

basis so your mileage will definitely

vary here Park District's so the client

that we're going to talk about is

Chicago Park District Park District has

been a client of Clarys for they think

six or seven years at this point the

folks at Park District's that I've

worked with I've probably worked with

them for close to 15 years because I

work with them and in another capacity

as well but a little bit of Park

District's background as an organization

tons of green space obviously tons of

parks many of those parks are along the

lakefront Chicago gets a lot of things

wrong but they really did a great job

a public space especially after the fire

but well almost more importantly at

these parks and facilities they're

offering thousands literally thousands

of programs and events every year just a

side note if you live in the city of

Chicago and you're not using what the

park district has to offer you're really

missing out especially especially if you

have little kids there's tons of stuff

to do with kids at the protesters so

they have a lot of content tons of

content and whereas you might think that

there's not a lot of content with a park

that changes all that much the opposite

is true they're changing their hours all

the time they've got different events

different programs things might be under

construction

it's information they have to let

everybody know and by everybody I mean

everybody they're really trying to serve

the entire metropolitan community of

Chicago across all the different ages

ethnicities North versus South they're

trying to do it all and they have to try

and communicate that primarily through

their website and their website does

take a lot of traffic the page views

were very tremendously there are some

days where they're only half a dozen

hits and some days where they're going

to have 500,000 those are spikes that

are really driven seasonally so for

instance what is today today is the 9th

probably two weeks ago probably one of

their most popular pages would have been

their ice rinks page this is you know

the end of winter everybody wants to get

out there I'm starting at about you know

a month or so the beaches pages and

swimming pools pages are going to become

very very popular so I'm telling you

this in part because you can't just

optimize one area of the site if the

whole site has to be really responsive

because depending on what time of year

it is they're going to get hit having

said that there are some pretty standard

pop

pages regardless of the season programs

parks events and jobs so there are a few

areas that we could focus on yeah so

what prompted a change we launched a new

version of the site at the end of 2017

took us about a year from start to

finish to get that sucker out there and

done but why do the move the biggest

issue was that the old platform and this

is a screenshot of the old homepage the

old platform excuse me was on an old

customized proprietary system and there

was nothing wrong with that system to

begin with I just want to be clear about

that but it had probably been customized

over the course of five or six years

four years maybe because of client

requests in other words kind of the Park

District said yes this is you know a

great page that we want mystery care

this tweak there so by the time Park

District was looking at replac forming

there was almost no one who understood

the codebase of this site so they really

needed to get on to something that was

stable and usable and you know could be

maintained because the old site was

really a case of you would make an

enhancement over here it's something

over here any website would break the

other issue in terms of moving was that

because this site had been built again

several years ago I think it was

originally built five or six years ago

and then they refactored the design to

be responsive three years after that

refactored meaning there were some major

pages in the site like the home page

that was made responsive but then there

were some interior pages where they just

kind of went yeah we'll do the best we

can but we're not going to rip out the

program so we did need to make it

responsive through and

so the goal was to replac form the goal

was not to do significant redesign

however it's really hard not to do that

and really hard not to take advantage of

a project like this so we did end up

doing some redesign which we hadn't

intended at the beginning so where were

we starting from we had a lot of data

Park District I think at this point has

probably over a decade worth of Google

Analytics if they really wanted to dig

back into it so we had Google Analytics

we knew what pages were getting hit they

had done some quick Google user surveys

but just like the standard widget so you

get to ask like a few generic questions

so now a lot can we you know that we did

do some significant benchmark and

comparison analysis and that's the

screenshot of what you're seeing here

which is much longer and much wider

Excel spreadsheet that we created based

on probably I think it was a six or

seven major comparable parks and then

probably another dozen websites that the

part district folks were interested in

basically to see what functions did they

have

what were the labeling things this was

turned into a really big deal when we're

going through this process were they

calling it apart were they calling it an

activity were they calling an event so

we had all of that so we did have a lot

of information to start with but it

wasn't really enough data only really

takes you so far because you can say

well this page always gets hit and then

this page likes a photo galleries never

gets hit but if you're just looking at

the Google Analytics you don't know why

is it because people are not interested

in throat o Galilee's or is it because

they can't find them this turned into a

whole series of conversations where

everybody's staring at the sink data and

guessing about what it meant so what you

really needed to do is find out how

people are really using it in terms of

finding out how people are really using

it it's also sometimes better and it

certainly was in this case to get

external validation because even though

practiced retired clarity partners for

our expertise at a certain point when

you're having these arguments you'll

you'll inevitably have conversations

like well you just don't know where

users the way that we know we're users

so there you know you get into these

situations we can't really resolve the

conflict a bonus for perk districts and

other organizations that we've worked

with

it's the publicity of using a focus

group if I'm doing a major project and

I'm a private company I can spend my

money how I want if I'm doing a major

project and I'm a public government

organization and it's a lot of taxpayer

money that's being spent you better have

a good justification for what the

outcomes are so they also had the

advantage of using the focus group to

say hey we reached out to actual Chicago

residents we got their feedback and we

incorporated it but the goal ultimately

using a focus group is find out how

people are using the site so this is

where a confessed that we had a secret

weapon then not everybody can use a lot

of people can which is part of why I

want to have this slide up here but we

were using smart Chicago that

organization excuse me made a major

change at the end of December of last

year and they were now known as City

tech but they exist their whole mission

is to improve lives in Chicago through

technology and the real secret sauce is

they already have a built-in audience of

testers

their civic user testing group pep group

look gorg will tell you all about their

methodology but the key here is they've

got literally thousands people already

signed up willing to test technology

projects that are put in front of them

so we've worked with them before on a

couple of other projects and again I

know not everybody can use an

organization like this which is what I'm

gonna talk about some alternative

options later so here's I want to get

into the nitty-gritty of what to expect

and what we went through

so park district clarity smart Chicago

Mets we clarify their goals what are we

trying to do and the goals really came

down to power people using the old

existing site how are they getting

information what's working what isn't

working so the first thing to figure out

is who do you want to test this is

important regardless it might seem like

an obvious thing but cast your mind back

to a few minutes ago part of the reason

that we were doing the testing with the

user focus group is we had these you

know Clash of the Titans arguments about

well why are people doing this you do

not want to invalidate your user focus

group by getting the wrong set of

testers so for instance I have been

happy to test I have been happy to be a

tester for this but I live in Evanston I

don't use the perk district resources I

don't use their program from their vents

you don't want people that that are

doing the testing that the client can

then come back and say well who cares

what Kelly thought I mean she's not a

resident so you get think about who it

is that you want to test in our case we

wanted to get people from Chicago

she had to be Chicago resident and then

they wanted a mix they wanted a mix of

people how to use the website and your

hadn't so they were getting a

cross-section people who had and had

participated in their events and

families and seagulls really getting a

across demographic the next item was

developing the questions which you would

think would be really straightforward

and simple and it wasn't you know but

the idea is to find out what are the

pain points of the website how can

people find information is it easy you

know giving some specific task oriented

questions but it took us a while to come

up with those questions but then what we

found out is once the first we had a

shared Google spreadsheet that we had

when we were starting to draft the

questions and once we had the first five

or six questions the floodgates opened

like oh my god there's this thing I've

been wanting to ask people for years now

I have a chance we ended up with way too

many we have over 50 questions and there

were 50 questions that we all cared

passionately about and Sonia who was our

main contact at smart Chicago was very

diplomatic and very gentle and basically

said no way you can't ask this many

questions

I think her original target was to get

us down to five or six so it became this

part of the process for us actually

became the most painful because we were

already invested in those 50 questions

that we'd come up with so to then go I'd

have to like kill your children down to

five or six was really hard so the other

thing that I thought would be useful to

share in terms of trying to figure out

how to eliminate questions is what makes

a bad question and basically they fell

into three categories there were some

questions that were too big one of the

questions we originally had was why

would you visit chicagoparkdistrict.com

what are you looking for when that

question just smells of desperation you

know I mean it doesn't really give

people a focus to hang

- we also have a question what is your

opinion of the photo usage on the site

which images are informative and which

are merely decorative that's try asking

somebody to solve an internal argument I

mean that's so clearly me arguing with

Aaron about our marketing photos you

can't ask somebody to answer that for

you some of our questions were too

narrow what incentives do you look for

before participating in a loyalty your

points reward program how would you go

about contacting a local park advisory

council those are actually legitimate

questions but if we're trying to have a

focus group that's solving the most

problems for the most people this these

are not it I mean especially if we're

trying to get down to a smaller set of

questions and then finally we have some

questions they weren't really solving a

problem how would you find and apply for

a job well if you remember the jobs page

is one of the highest hit pages in the

site already people are obviously

finding that likewise how would you

register as a vendor if you wanted to do

business with Park District I can

guarantee you that if we buried that

page seven levels deep with no

navigation access to it the people who

want to do business with practice are

going to find those pages anyway so

those weren't really solving a problem

so what makes a good question ultimately

we narrowed it down puts a fewer than

ten which again was really really hard

for us but they were task oriented they

were task oriented and specific and

trying to be as close to something that

people would use as possible one of the

things that we ended up doing is the

focus group ended up being in

excuse me one of the local libraries so

one of our questions was search for

Chicago Park that's closest to this

library find the hours for an indoor

pool closest to you and then we decided

to throw in a few sort of trivia hunting

questions like finding out the year the

Buckingham Fountain was installed I do

have the URL for the final report here

excuse me

and after mid camps over and we upload

the PDFs to the site you can grab that

one of the which I should have said

before one of the mandates for smart

Chicago is they are happy to work with

different organizations but one of their

requirements is they have to be

completely transparent so if you go to

that report you will see all the

questions we used the summary report

that is smart Chicago provided and then

the final full report so if you want to

see all the questions in there sort of

as a good guideline so what was the

process like scheduling was slow from

our initial discussions to actually

doing the test itself was about six

weeks and then I think it was probably

two weeks after that for the summary

reports come through I'm mentioning this

because if you're thinking about doing a

user focus group and at least a user

focus group that's giving you feedback

on an old existing site the idea is

you're going to use that to inform how

you're going to build the new site so if

you're going to get the results back it

still has to be in the discovery process

of your project otherwise it's no good

in other words it doesn't do you any

good or it doesn't you belong money I

suppose but normally it doesn't do you

any good to get this sort of feedback if

you're already full month into

development because if there's anything

that comes back that says oh the

navigation that we thought it was

fantastic is new

for anybody are you gonna throw away

that month worth of development start

over you're just gonna say well I guess

sucks to be a user of our website and

again you can scale these things as

large or small as as you want to which

again I'll talk about a little bit but

do know that there's you know some

effort that you've got a plan for

initially so there's the you know the

usual like one is everybody available

there's also in terms of scheduling the

actual focus group itself just the

stupid physical space like where are you

gonna be we ended up having I think

ultimately we had a couple dozen people

do our testing we have more respond but

the day of I think it was a couple dozen

so we have space for that and then

equipment it's just a question right I

mean are you asking people to bring

their laptops or are you providing

laptops for them you just need to plan

it out in advance there is then also the

issue of getting the people getting the

participants so there's the time lag

that you'll need if you're sort of

sending out emails and scheduling with

people that you don't know which was the

case here there's also getting people to

help you Proctor the test itself we

ended up having the settle later slide

we we ended up having five smart Chicago

staff and then five other folks either

from clarity what from Chicago Park

District to help us with this which I

will tell you at the time I said well

it's a lot of people it was barely

enough for the testing that we did so

you'll need to make sure that you're

getting enough people to help you again

depending on the scale of what you're

doing

crucial here is this is all possible to

get a client representative to be part

that test with you so much better we've

done this a few times we've had a client

be with us and also the situation where

the client wasn't with us you can

summarize your findings you can provide

a report and I will tell you that if

you're providing report dry or not dry

depending on your talents the writing

this and your report says navigation and

the site is really difficult for people

with visual disabilities and here's

where it's particularly problematic that

is not anything is impactful as having a

client sit down with a woman who's

visually disabled and asked her to find

a park and listening to her difficulties

and having to heard having to repeat

over and over and over in searches which

was an experience that we had and I will

tell you about twenty minutes into that

I think Aaron was ready to burn down

this site and just say you don't have a

website we're running away this is too

awful so if you can get a client to

being part of the focus group it's huge

because they'll have an emotional

response that they won't have otherwise

so the next thing we did is you know so

we've got our time schedule we've got

our testers we figured out the logistics

for these these things and then smart

Chicago said okay we need to have you

guys come in we're gonna help run this

test and we'll do a review of the

procedures now at this point I was happy

to do it

I did have a little bit of an internal

iral I'm like what this movie's

straightforward I don't think we need to

do this thank goodness we did first of

all going through the review of the

procedures for the actual tests I think

put the rest of us at ease

just going through and saying okay we're

gonna have 28 people come through that's

how many confirm but again I think we

only had a couple dozen everybody's

gonna have roughly at 30 to 45 minutes

slot it's over this you know three and a

half hour time so we sort of a constant

rolling through of people when someone

comes in last them to sign a form we'll

sit down with them we'll assign two

people to be with them and by the time

Stoney had run through that I was like

okay

I get it that seems good and then what

she did with which I thought was really

smart as she explained okay this is how

you're gonna run through the test you're

gonna ask your users to stay out loud

all the time what it is that they're

doing you're gonna be capturing what it

is that they're doing but at the same

time you the proctor of the test should

really be keeping in mind what you think

the user is doing that they haven't sent

out loud so you can make your notes

later on so you're you are being asked

to do sort of a a double think while

you're doing this if you're a proctor so

we're like okay yeah yeah yeah yeah and

then she had his pair off with each

other and run through a few questions

which was highly revealing because I

realized wait okay I'm typing at the

same time that I'm listening to what

this user is going through this is

really hard

so prepping for the test even if it's

just a few minutes to go through how are

you going to ask these questions really

really useful so the test itself did

give us some surprises that I wasn't

expecting and this is sort of again what

I thought would be useful to share with

other people physical space matters so

we had a nice room

a local library is about this size with

lots and tables and chairs

I think our tests ran from 4:00 in the

afternoon to 7:30 something like that

which was great in the sense that you

know it's fairly convenient for people

who are getting off work and kind of

crappy in the sense that most of the

people who are coming to the library ran

into traffic issues because we were

asking them to come basically during

rush hour so what it meant was our plan

we had everybody designated with you

have this 30 minutes line you have this

missing their appointments and stacking

up so we thought we had plenty of space

and we kind of didn't at the end of the

day the other thing that we ran into

where physical space matters a lot is we

stressed the hell out of that library's

Wi-Fi we hadn't really thought ahead by

having all these people because you know

the testers are on laptops or phones

whatever we provided and then the

proctors are also on laptops or phones

trying to take notes and there were a

few times where you sort of have this

ripple go through the room with like

five so physical space matters again

just keep it in mind when you're

planning your own testing so we had

these 30 to 45 minutes lapse which again

I thought in my head like 10 questions

come on under way too much time I don't

know that any of our tests actually

squeezed in two minutes the one test

that I've already mentioned that I was

doing with with one of the testers who

was visually disabled I think we were

with her for almost an hour

running through those questions so give

yourself enough time it's going to take

more time than you think in part cuz

people get really really invested in

these tests the other thing is even

though this test we're taking so long

we're taking notes the whole time really

trying to keep up we could barely keep

up because again people are invested in

really trying to do their best to tell

us everything that they were doing the

it was just awful they're revealing

sitting with Maureen for over an hour

really made me invested in missions sure

that the new site was in better shape

one thing that may or may not again your

mileage will vary that we found was

really frustrating for everybody

involved is the users got confused over

different sites our mandate was to find

out how are people using dub dub dub -

time apart district common not

surprisingly Park District has all kinds

of other external sites that the dub dub

dub site links to it wasn't we didn't

really care about the purpose of our

test but if the user is going off on the

park point site and spending 20 minutes

there or going off on the registration

site and spending twenty minutes there

it was really hard to just explain to

the user no go back to where we were

before because again you don't want to

direct that much of the users experience

it was also really really hard not to

help the people that were testing I mean

we had these task oriented questions

they were kind of set up like a

scavenger hunts in a lot of ways so we

knew where people could go to find out

what your Buckingham Fountain was

installed we knew where people could go

to find out the indoor pool closest to

them it was really hard as usually we're

fumbling through not to say

thank you what made this especially

difficult is we had talked about in our

preparation for the test we talked about

you know if users get stuck it's okay

you know they don't have to find the

answers to these questions because our

interest is just in their experience of

trying to get there what none of us had

really banked on was the people who were

testing really didn't want to be wrong

you really really wanted to find the

answers to these questions so even

though we would say it's okay you don't

have to find an answer it was I don't

know I really want to find miss Doane

Park which also made you know the tests

go a little bit long but users being

sensitive to being wrong was just

something we were not expecting or

prepared for all right so let's talk

about outcomes so as I mentioned we had

the test like all those people come

through they were great it gave us a lot

of good feedback and about two weeks

later we got the summary report back

from smart Chicago it answered a lot of

our questions and also gave us the

ammunition to make some changes so you

will see excuse me

another screenshot from the old homepage

on the left you'll notice I mean there's

not a lot of changing information that

you'll notice on that left-hand

screenshot there is a lot to look out

there

you know one of the questions that we

had for our testers was you come to the

home page what is it that you want to do

so Sonya when she was providing the

summer and report back to us again very

diplomatic

very gently when she was going through

the percentages of what people were

doing she said I have never seen as

fractured a set of results as what you

guys got when I'm asking people to go to

a home page you know some people

I clicked on the big green banner some

people are happily looking at the things

next to the hero image that we're going

to the news it was just all over the

place

we had already been arguing suggesting

strenuously to Park District that they

really needed to clean up their homepage

they really needed to give people more

of a focus on what to do

and so the user focus group results

really allowed us to push that argument

further and say you really really want

to clean homepage yeah I will jump ahead

a little bit to the end and say surprise

surprise this argument is not over

because once we launched the site some

of the feedback we got was well it's too

plain the homepage is too plain we need

to move things up we need to have more

things around here so the real estate

battle continues but at least initially

we did get to clean up the homepage for

them that was directly out of the user

focus group results the other thing that

came up very very clearly out of the

focus group was you guys got a focus on

mobile I mentioned before that the old

version of the site had optimized some

pages but not all of them you will see

in particular this notice retracted on

the search page on the old site was

really really cluttered and almost

unusable on a phone so we were able to

focus on mobile streamline searching so

this is where it's talking about is the

screenshot on the left which is the

whole site but just kind of gave up on a

mobile device there wasn't much you

could do there so instead of having it's

a little bit hard to see but instead of

having

you know all these different options and

dropdowns that would expand the feedback

that we got really clearly from the

focus group is you got to make the

searching simpler so we collapsed

basically the the options that people

had

based on that feedback highlighting the

mapping options so this was interesting

again it's a little bit hard to tell

with the old screenshot on the left but

there are these tabs indicators that

second tab is a link to a map of where

all the swimming pools are no one ever

saw that and we were gonna play through

that the user focus group they never saw

when we're asking for map or geographic

related questions that there was even an

option to go to a map so one of the

things that we did with the redesign is

really make the mapping option front and

center for people so you really couldn't

miss it so all that's great what if you

don't have an established Civic user

testing group available you can still do

your own focus group you really can so

these first few bullet points that I

have here clarify your goals develop

your questions we didn't need cut

briefly at certainly not for clarifying

our goals and developing our questions

we really just needed somebody to say

you can't do a user focus group of 50

questions I'm telling you now focus

group with 50 questions just pare them

down develop them as pare them down

and focus find neutral testers that

doesn't have to be you know going out to

residents of the city of Chicago we have

done user focus groups where it's it's

literally five other people in the

office now they're still fitting the

demographic that we're looking for for

the particular project and I'm thinking

of their worst acog o residents they

weren't in any way shape or form related

to the client or the project that we

were working on and but they were you

know available they're invested and any

testing is better than none this is an

old an old particle link from the

Nielson group but it's still really

valid which is five is usually enough

you know we have the luxury of having a

couple dozen testers it gave us the

luxury of having a really broad across

demographic section of people but if

you've got five people testing you're

probably gonna get and this isn't like a

real stat this is just my anecdotal stat

you're probably gonna get like 80% of

what you need from five people after

that you know you're getting little

cherries on top but after after five you

got most of what you need the incentives

don't have to be that big cut group if

I'm recalling correctly they give their

users a five dollar Visa gift card to

sign up to make themselves available and

then for participating in a testing I

think it was a 25 dollar Visa gift card

so it's not overwhelming in terms of you

know what your ultimate project budget

is gonna be and then finally just spend

a little time prepping for the test you

know there's going to be probably less

prep that you would have to do if you

don't have as many users and you don't

have as many logistical difficulties but

you know grab some

and just test one through the questions

for five minutes 15 minutes you'll

inevitably find a way that question

sounds kind of stupid now but I'm saying

about or oh wait this is actually gonna

take my user longer to figure this out

so I need to buffer some time so as long

as you prep for the test you should be

good so what's next for us I'm hoping

and what in discussion with city tech

now I'm hoping that we can go back and

basically run the same test with a

different set of users to basically

validate did we solve our pain points

you know if we're looking for a redesign

Reap lat form in order to get users to

information more readily more quickly

did we do that or are we seeing the same

issues over and over again we haven't

planned that out but we're hoping to the

other reason other than just validating

our pain points which I've already

alluded to before is we're still

fighting about real estate we're still

fighting about well why aren't we having

the news more comment on the home page

why don't we have 5 more items in the

IRL navigation because everybody wants

their piece of that home page or their

piece of that navigation and when we're

coming across these conversations and

we've come across quite a few of them at

this point we're still having this Clash

of the Titans things where okay we need

to get some users to validate this so

the other thing that I would leave you

with is this isn't necessarily like us

with so many things with websites this

isn't necessarily a one and done kind of

thing it may be you know something

I want to do periodically again I can

say that a little bit of a luxury with

Chicago Park District that's something

that that client is invested into they

want to get user feedback they want to

know how people are using their site

again that that will vary per per client

all right I ran through that like an

auctioneer I'm afraid that I wasn't

intending to but any questions yes do

you ask customers to bring their own

devices and not you

she's the browser or how much materials

so for the tests that we did with

Charlotte Park District we provided that

we snatch Chicago provided laptops and

iPads particularly with the disabled

testers that we had iPads with it's

voice over but some you know built-in

recognitions the other thing I should

take to is part of the reason that smart

Chicago was providing their laptops as

they already had built-in screen

recording devices screen recording

software so it wasn't just should've

said recently it wasn't just the proctor

scrambling writing their notes when they

were developing their report later they

were also able to see the recordings and

what people were doing solids were

different we did ask people to bring

their own phones if that's the device

that was being tested but that was

determined ahead of time so for instance

when the smart I were sending out their

questionnaire there's there basically

ask me would you be willing to bring

your phone to test and if they said yes

then it's much cago it said okay here's

your time slot that you're coming on

Tuesday afternoon would you please bring

your phone there's even a divorce and

exhaustion in other much smaller

settings have we done we just asked

people to bring their stuff

did you use a camera or something to

record the phone with tests no we didn't

have any recordings for the phone test

if I recall correctly so that was a

missing piece for that for sure so in a

case like this you know you'd ask

somebody a question about Buckingham

Fountain yeah you're gonna get data back

either it's easy to find or it's

difficult to find on the site like this

where there's tens of thousands or

hundreds of thousands of bits of

information rather than like we do in

your entire you know how does this play

into not everything could be on home

page so you could have a good navigation

structure give it a good search

structure but how does that one little

piece help you so that that one little

piece actually helped in a couple

different ways one it was and it was it

was a prototypical example question

right one was how are people even trying

to find the information okay so are they

even starting from a point of I'm gonna

use the navigation or are they starting

from a point of gonna use a search so

there were several questions that were

like that which is like it is navigation

even an element in somebody trying to

find this the other thing which was an

unexpected outcome for some of these

questions and that one in particular was

we discovered that Park District wasn't

keeping kosher with its content and

they're actually at least two and I

think three different pages on the site

about Buckingham Fountain that had

slightly different information on than

those pages so that was an unexpected

outcome basically my back to part two

servings of stop stop doing this madness

thing that you're doing but it was

really that first item that I mentioned

it wasn't more important which is how

are you been trying to get there so for

instance just to close the loop if

you've got

people who are in charge of special

artworks or special elements that Park

District owns that are saying like I

want you know Buckingham Fountain to be

under the drop-down under you know parks

you say listen people aren't even

starting there so you're just taking a

very valuable real estate when you don't

need to

we rely on some logic hago entirely for

that reporting and their mandate was as

many very system very clear as an

organization all the way through that

everything is completely transparent so

in addition to the URL that I provided

before with that sort of main page there

are the questions that we use the

totally wrong results from the testers

and Proctor's I should also say when the

the testers and Proctor's are recording

their notes the testers get to use

whatever

thank you many want you know so what

you're seeing in that report is you know

user Red Dog 2020 as this response is

still anonymous in that sense and then

smart Chicago took that totally raw data

they went back and layered their

analysis on top of it provided that

summary report to both cleared and Park

District and then they clean it up as a

final report that is also public on the

site it was great to have smart again as

another advantage of using an

organization like that it's ready to

have them do the summary report because

then you can honestly say that this is

an organization that has no investments

in these design arguments that we've

already been having for eight weeks so

it really again gave that income on tour

of neutrality to it

the first is he mentioned

through ours

they only say

really both both so we talked about this

before and I forgot to do it

let me repeat your question so the

question was as users are being asked to

say out loud are they saying what

they're doing but also are they saying

what they're expecting and if any

implicitly are we recording matter yes

yes and yes so those part of the

scramble right is when we're asking

somebody to find Buckingham Fountain and

they're typing in to the search bar and

they're frustrated with the results that

are coming back you as the proctor

saying okay well they went straight to

the search bar they didn't try to do the

navigation at all they just went to the

straight to the search bar they're

saying they're frustrated they're saying

what they're expecting and you're trying

to record this I will say that again as

part of the preparation process that

Sonya put us through and we realize even

then that it was hard to keep up she's

like don't try and write complete

sentences don't want time for that just

write something that is sensical enough

that you can come back and understand

generally what the user was trying to do

and to actually you know verbalize what

they're feeling

they're inexperienced or there

and maybe shy about it or they're not

sure which

how are you could sound like they were

really talking about so how are you they

were but you're right so that so the

question that you're asking is how do

you get users to verbalize what it is

that they're doing or trying to do if

they're not inclined to verbalize that

that was also part of the preparation we

did which is Sonia was asking us to ask

questions but it was also you're right

it's hard not to ask questions without

being leading but I think that was part

of the advantage of having really task

oriented items for them to begin with

because what we were they what we were

asking them to do was already fairly

neutral you know finally near the

Buckingham Fountain is installed there's

nobody's invested in that really well if

if we got a user that was and this is a

strongly an example but if we got a user

that was silently going through a couple

of navigation items and then going to

the search at that point you know I

would intervene not intervene but say

out loud why did you decide to go to the

search at that point and then get her to

say what thought it was gonna be here I

was gonna be here and it kind of gave up

we were just meant to search I have done

focus groups that have had the pitfall

that you were just talking about we did

a focus group with another clients a

year and a half ago and part of what

we're trying to do with that focus group

was the client already had a website and

they were internally trying to decide if

they wanted to have a website or mobile

app and unfortunately my boss came along

through that focus group and he's a

great guy by the way I just to be clear

but he was so impatient

we were like 15 minutes into this and it

just lunged in because like if you had a

mobile app for this because then you

could do XYZ

and honestly like half the room was like

no we don't want to a mobile app and

then he went into a sales pitch I'm

Gerald and I were sitting there going

well okay so much for this focus group

seven test results we try again with the

next group but yeah that fix that you

don't want to be leading at all possible

yes

what

what time since it is a very Jewish

website where there are no people of all

kinds choosing this what kind of

demographics did you specify by choosing

this because you know when you talk of a

person with disabilities or something

like that that kind of user doesn't come

across thank you

so we kept it pretty broad so the

question was what kinds of users did we

specify for this we kept it pretty broad

in part to be honest with you to take

advantage of the pool that smart Chicago

has meaning smart Chicago has done a lot

of effort and outreach over the years to

get an ethnically diverse set of users

they don't have the diversity that

they're looking for so we didn't we

talked about we didn't want to throw

that in as part of the mix because they

were afraid we weren't actually going to

get that many users back what we were

comfortable throwing in as part of the

mix is getting users from different

parts of the state so we did ask we we

just asked like what their zip codes

were people know they already had that

when when you sign up you have to tell

that so we did tell us much Chicago has

it for instance we don't want everybody

from the north side you know we gotta

get sort of a range of people we also

asked for not an age demographic but

were you single or did you have a thing

because the other thing that Park

District cares a lot about is serving

kids so we wanted to make sure that we

had people who were if they were already

interested they would be interested in

signing their kids up for programs and

that's really kind of broadly where

we've stayed

we didn't go deeper with that did you

analyze the demographics afterwards they

did so you'll see in me well again with

I constantly think yes we did within

that really kind of broad set of buckets

yes smart card oh did that analysis for

us so and that is part of the summary

report that they have available to be

honest with you and they don't know to

be pre fractured there wasn't there

wasn't a clear indicator that if you

were you know a parent or not a parent

you've got to fucking completely Foster

honestly the only difference was if you

if you were a parent with kids you are

doing how to register for the programs

but that was kind of it in terms of the

demographic usefulness they've got out

of it

you can forgive her this this group that

you have private esters that yours a

minimum sort of technical skill amongst

the testers I mean ah who didn't so that

the question was was there a minimal set

of tech savviness skills among the

testers I am trying to honestly remember

I don't think that that was a specific

question that we asked but I think that

it it's a self-selecting group because

smart Chicago again there their mission

is all about using technology to make

life better in the city of Chicago

so these testers are always testing

something that's technical it's not

necessarily website could be napped

could be something else so if you're

willing to sign up for that you're again

you're already self-selecting to a

certain extent which is good and I park

districts as you may or may not be aware

has these seasonal registrations four

times a year in the Park District staff

sets up a temporary call center which

I'm part of and so I'm there answering

questions and again if you haven't

experienced that these registration

periods are insane it's like a nine

o'clock on a Monday morning all over the

city parents are poor days they've got

multiple browsers multiple computers

going to like sign up their kids but you

get these calls and they're kind of I

don't know surprising but also

refreshing I just did this a couple

weeks ago and the woman that I was

talking to didn't understand that a

highlighted text was a hyperlink that

you

we don't get those people coming to our

test but they're obviously still have

their engine is detailing for were you

then able to use hindsight to look back

at the Google Analytics and find that

maybe Diller indicators of some of these

findings already I mean so the question

was having gotten the summary report

will be able to tie that back to the

Google Analytics fundamentals but no I

mean I can confidently say no we didn't

do that but I'm trying to think if any

of the questions that we had could have

tied back to that I'm not sure that

could have so again the buckingham Park

example we could certainly have seen

well how many pages how many times at

the Buckingham Park page ends plural

being hit how many times is Buckingham

Park being used as a search term but

that didn't give us the insight that we

got during a test of people didn't even

try and go through navigation they just

went straight to the search term so we

still it's still a different set of data

at least at least for this experience

yeah I think your point is well-taken

which is in a different instance you

could do exactly that

just not for us any other questions

more questions the intercom plenty of

the reports it just could you talk a

little bit more in general about how you

make certainly choose the large amount

of data that's because I haven't been

involved in project leaders have all

these notes from user testing so so the

question was you're getting this

information back from the user testing

and I would expand that actually to get

all this information from the analytics

and the benchmarking that you're

starting up with - how do you make sure

that you get it used broadly not all of

it was used it just wasn't in the report

so in it in the presentation that I had

I mentioned we've seen a four major

outcomes that we serve clicked onto the

report itself had if memory serves

something like ten like specific

findings once we got past those first

four they were either fights that I

wasn't willing to take on because I

wanted to get the project launched or

they weren't important enough to really

make an impact but that was true for

those other things too meaning we did

this benchmarking which is a pretty

extensive process and probably

especially comparing other Park

District's and what they were doing

and I think probably out of that I'm

trying to remember Katie anything is

like a 50 page report I think out of

that there were probably only three or

four findings out of that that really

made their way into the final redesign

week platform but that's okay you know I

mean not everything is gonna get in

there you can't win every battle I think

it's okay if you do

these efforts these focus groups this

you know research and even if just one

or two gets back in I so called a win

making decisions with data is thoughtful

reasoning always alright