Visual Regression Testing - what are people using?

DAVID NEEDHAM:
OK, there you go. So I originally called this partly because we...it seemed like there was not very many topics on the board to discuss. This is a topic that I've been interested in and involved in teaching for four years or so. And I'm just acutely aware that there are many visual regression testing tools out there. There's many options for people to do things like this. And I'm not sure if Backstop JS is still the hot new thing. I don't know if it's still super actively the thing that people pick for problems like this. And so I wanted to just, I guess, propose the topic, but then probably sit back and listen to see what things other people are talking about in the space and interested in hearing more about. So that's why I'm here as an introduction, would someone else like to jump in?

AMY JUNE:
Well, I wanted to come because I don't actively...I am a QA engineer in my job role with Canopy, but I've been sort of taken off of that role recently because I work more in the community. And we're focusing on that because of just right now, that's all the events are happening. But one of the things about the QA engineer role at my shop is we try to allow it as a role like a transitional role for folks like who don't have that coding web development within our internal agency but are looking to move in. And so we sort of thought that the QA role was a really good role for them to be able to introduce themselves. So the back end and learn how Drupal works and things like that because they're looking at architecture maps of doing visual regression testing. But one of the things that we use almost exclusively because we do have like the junior coders or the people who aren't coders. One of the tools that we use is Backstop JS because it's so straightforward. The documentation, all the knowledge base information is so easy to understand that anyone can kind of jump in and create a Backstop JS file and with the help of like maybe a developer integrate that CircleCI.

So I was coming in to see what other tools people are using that are that straightforward. So internally within Canopy, we are using Backstop JS on all of our projects. We integrate them with the Pantheon workflow. We have multiple dev set up. It's just because of the ease of use. You know, really it's just one file and you're comparing URLs and you scrub. So that's why I'm here.

SPEAKER:
Yeah. I'm at the end and we use a few different. We started with Cypress for our testing and then I believe we switched over to Nightwatch because (UKNOWN) I can't run Cypress in the environment for whatever reason. We are...we've used Nightwatch recently and then we also have used...what was the other one? I forget, it escapes me, but Cypress and Nightwatch have been our two most popular that we've worked with. And then we use, you know, Percy for dating snapshots and whatnot.

DAVID NEEDHAM:
Well, thank you for sharing. I'm trying to add in notes as we're talking. So definitely review update, edit things that I'm getting wrong or are incomplete if you see anything. But thank you for sharing that. That's...it's always good to get some new perspectives. Some new options. Welcome, Kim.

KIM SARABIA:
Alright, thank you. Sorry, I'm late. I had to type up some notes from the last session.

DAVID NEEDHAM:
It's OK. So we're just giving introductions, really, just like sharing what we were hoping to learn about here and maybe something that we've done in the past, what tools we're using right now, if anything, for Visual Regression testing or other tests.

KIM SARABIA:
Michael mentioned Percy, that's something that I've used. And really, to be honest, that's about it. That's why I'm here because I really need to figure out a way to do that for my current projects. I just started at my current job last September and we don't necessarily have something integrated into any of our workflows yet so that's something I'm trying to get done. But I'm wanting something quick and easy, but I don't know if that's possible.

DAVID NEEDHAM:
Yeah, interesting. Percy is one that I've never heard of before. I'm looking at the website right now.

KIM SARABIA:
Yeah, it's pretty neat and that's quite costly, but I thought like we basically saw them do a demo couple of years ago in JS camp. And, yeah, I mean, it's very straightforward. I don't even (INAUDIBLE) the extensive development experience to actually use the interface. But and I'm sure Michael can probably say more than me, but it basically will just do screenshots of a settlement of pages that you wanna QA and it will do a side by side comparison. I don't actually know what's going on in the, you know, behind, you know, internally, but I do know that they will do like an actual...they can present like a side by side comparison of a before and after you push into a branch or create and you commit.

DAVID NEEDHAM:
Cool. Yeah. Interesting.

KIM SARABIA:
Yeah, the downside is that it's I think after 30 snapshots, they start charging you so.

AMY JUNE:
I went to the website too and I saw that they're part of the browser stacks, is that...is it within the testing tool or just that they're bought by them? Does anyone know that? I can't tell why.

KIM SARABIA:
Last time I used it, it was we had integrated into...I'm not sure exactly, but it wasn't through browser stack. It was through our workflow. And I think Jenkins was involved. So, yeah, it wasn't through browser stack though, I can say that much.

DAVID NEEDHAM:
It looks like they were just acquired, but I haven't logged into to see.

AMY JUNE:
That's what it seems to me and I just wanted that clarify maybe (INAUDIBLE). (INAUDIBLE) browser's stacks but there's, from what I can tell, no Visual regression testing involved in that at all we know so.

DAVID NEEDHAM:
Yeah, it looks like from their blog post, they're saying that "Sometime in the future, there might be integrations between the two, but right now, nothing is changing." You know, Nightwatch is one that I haven't really heard too much about either despite Amy June, you were asking on Slack somewhat recently if we had folks at Pantheon that were sort of on top of Nightwatch, but and I saw that there were sessions yesterday, I think. And they're getting started.

AMY JUNE:
Someone, I don't know if it was you or who else did it suggested Matt Gleeman. And 'cause I searched the Internet for it forever and I didn't see any sessions at any Drupal camps or word camps or anything. And so, yeah, he did like two sessions one like how to get you...how to get it working locally. And then the after lunch was specifically how to get it working with Oliviero so they could springtown it on a Saturday. And I didn't get to go to any of the sessions 'cause, you know, organizing stuff happens and you just (LAUGHS

DAVID NEEDHAM:
Yeah. Well, do you know if those were recorded?

AMY JUNE:
They were recorded. I don't know if they'll be listed, but I can let you know that. But they were recorded. And we're gonna ask Matt to do a kind of training on Nightwatch at Drupal Camp, Asheville. I don't know if that will be of interest to him, but now that he's sort of given that live demo and got a feel for it, you know, if he's interested in doing it again more formally. That would be really spectacular.

DAVID NEEDHAM:
Well, just to potentially spur on more discussion, just to point out the difference with my experience with Backstop JS and what I see with like Percy, in Backstop JS is something that you install locally on your computer. You know, it's, I think, an open-source project. You just run it and it does it stuff. Whereas Percy and several other options that I've seen online, allow you to do like a...like software as a service. You know, it's all built-in. You get a shiny interface for picking and choosing, you know, what to test, under what conditions and all that stuff. But you end up paying for that. Not to say that one is any better than the other, just pointing out a difference to try to spur on a discussion.

KIM SARABIA:
What are the price points or tiers for Backstop?

DAVID NEEDHAM:
It's free. It's just free software you can install on your computer. But you have to use the command line and, you know, do a little...there's a tiny bit of coding to get in and start configuring it.

KIM SARABIA:
OK.

AMY JUNE:
But minimal coding like a couple of line changes and, well, at that...I guess that's...that that might be different for different people. It's about perspective so I should take back the minimal so.

KIM SARABIA:
OK. Good to know.

AMY JUNE:
And on the Internet, David has some really nice blog articles and links to some videos that are really straightforward and easy to follow and it only takes like a couple of hours to really get started with it and to kind of explore it and just click a button if you wanna like just mess around with something one day. It's really straightforward documentation so.

KIM SARABIA:
OK, cool.

DAVID NEEDHAM:
The link to that is in the docs or the short documentary we're working in.

KIM SARABIA:
That's the one. That is you website. It's your personal site?

DAVID NEEDHAM:
Yeah, I embedded the video from the last time I gave that presentation.

KIM SARABIA:
Nice. OK. Good to know. I see it now, yeah.

DAVID NEEDHAM:
It looks like Backstop has workshops available on Udemy now too.

KIM SARABIA:
OK. Yeah, that sounds...that's really helpful.

DAVID NEEDHAM:
I mean, it's one of those things where, you know, there's lots of tools out there that you could do yourself and set up locally and, you know, go through and put all the effort into learning it and doing it and stuff. And then there's the turnkey solutions where you pay a little bit, but all of that is hopefully easier, kind of obscured so you can just focus on what you're trying to do. So I think there are pros and cons to, either way, letting on who exactly is gonna be using it.

KIM SARABIA:
Yeah, I like that it's completely free because, yeah, I mean, it's not really something that's a top priority so it's one of those things where if it's just something we can try to do when we have time or we can carve out space for it, yeah, that would be really useful.

AMY JUNE:
We use it every single time we do a update, you know, just, you know, a Drupal update, security update or module update because of the ease of use of just adding the line. You know, just make sure that we do a content freeze. And then clone the databases down, you know. So the databases are the same on the whatever 'cause you're comparing two (UKNOWN). And we compare like it whatever branch that we do, the updates onto the...to the production site. And we just, yeah, freeze content for about an hour and then we get a, you know, a little bit of report. It's pretty nice, you know, and some of the bigger projects that have it running on CircleCI which is a little bit out of my wheelhouse knowledge to share, but it's totally doable. And so a lot of them, they'll add their changes like when they're just doing like feature requests and things like that. And it'll kick back that it didn't pass the test, you know, so they know that they have to start again. So some of it eliminates that. That human involvement at the development level which sometimes we can't afford, not price-wise, but staff-wise.

I mean, sometimes you cannot. So you don't...you have to get it done in a hurry and you don't have someone to do your review for you so.

KIM SARABIA:
Right.

DAVID NEEDHAM:
Michael, I see you asked a question in the chat. I don't know what you mean by non-headless mode.

SPEAKER:
Well, we'll need Cypress. If you run some tests in Cypress, you can turn headless mode off where it will actually run a...open up from a phone browser for you and run your test for you. But it will show it visually so you can see the webpage loading and it kind of gives developers an easier way to debug their antenna and test. And I like that about Cypress the most. So I was just curious it's Backstop had that open.

DAVID NEEDHAM:
I wanna say yes. I...there are definitely ways within Backstop to select the browser you want to use like virtual or I think actually, you know, picking one of your existing browsers. I think the default is that it will pick a virtual browser like premium or something like that. And so it'll all run under the hood and you don't see anything pop up. But I think there is a setting to make something actually show on the screen to debug as you're going to. I...that's not something I've done before.

SPEAKER:
OK, (INAUDIBLE).

DAVID NEEDHAM:
Alright, well, being in unconference session, this is done when it's done and it starts when it starts and we should feel free to get up and move around if there's nothing more to talk about here. That's totally OK, I just wanna make that space available and deliberate about that. I'm happy to stick around and keep talking about stuff. But if there's nothing more to ask or other related topics to bring up, then, you know, we can feel free to go to another session or take a break for a little bit.

SPEAKER:
Thank you, David.

KIM SARABIA:
It sounds good, thanks, David. That's really...I mean, I've learned a lot from the short time so I appreciate it.

DAVID NEEDHAM:
No problem, thank you for coming.

AMY JUNE:
Yeah, it's...thanks, David for putting it on the schedule.

DAVID NEEDHAM:
No problem. Thanks for coming in and contributing.