Using the Robo Task Runner to Improve Developer Workflows

MARK DORISON:
Good morning. Welcome. This is what we're talking about. Using the Robo Task Runner. I'm Mark Dorison, CTO at Chromatic. Chromatic has been working with Drupal building complicated Drupal sites and Drupal adjacent sites for over 15 years. And that is what I get to work on. We're starting off with a warning. There going to be a number of code slides in this talk. The good news is that it's purely to show concepts. You don't need to read every line or copy anything down. Hopefully I've done a decent job of making it very readable. If you see a slide with clearly too much code on it, it will quickly move to one where we zoom in on the important areas. So don't panic. And all but a few lines that you'll see today are in public repositories that I'll mention. So, if you're interested in looking more closely, you'll be able to check those out. So what's the problem that we are going to talk about today? If you're like me, you like to automate things and you've worked on a lot of projects that end up getting festooned with Bash Scripts.

They add up. And lots of people love Bash. I use it in situations where there's not, I can't assume that there's another runtime available. It's great for that. It's relatively ubiquitous. But if we're working in a PHP project like probably most of us here are, I'd rather use PHP. If I'm working in JavaScript project, I'd rather use JavaScript. I want to kind of keep things correlated with the language of what we can expect when someone's going onto a project. This is a PHP project. If you want to work on some of these automation, some of these, some of this tooling it would be great if it was in the same language. So, what do we need? We need a PHP task runner if we're talking about a Drupal project. And in this case Robo to the rescue, we're going to talk about Robo. So Robo calls itself a modern and simple task runner inspired by Gulp and Rake aimed to automate common tasks. So, you might already be using Robo and not know about it. Maybe you've heard about this tool. Drush is built on top of Robo.

So it's likely if you're using symphonic Drupal and you're using composer and you're pulling in Drush, you already have robot installed in your project and you could make more direct use of it. Robo was first included in Drush in 2016. It first shipped in Drush 9.0. That feels like it should be relatively recently, but Drush 9.0 came out seven years ago. Drush 12 is being prepped for release right now to put some context around that. So a lot has changed in Drush since 9.0 and Robo has been there for all of that. So some of the use cases that you can use Robo for. Certainly not limited to these, but automation, testing builds and deployments, backups and restores. Anything where you currently have a complicated bash script Robo could be a good replacement for it and probably more stuff that you're not doing yet. It gives you better access to PHP packages since it's written in PHP. If you want to, you know, if you need to make API calls out to something, you can pull in Guzzle. I think we'll look at that a little bit later.

And anything else like that, you can pull in all kinds of stuff via composer just like you do for your Drupal site. You can then use it in your tooling. Better readability both for developer and for the user and the logs. For the user running the calls it's a lot easier to give great output to the command line. Better error handling, better developer experience and user experience. So to sort of set the table, I'm going to show some of the tasks that are built in to Robo. This is some of the easy, like, things that it gives you out of the box that it's going to make easier for you. That said, you can also do anything that you can do in PHP. You're not limited to any of this stuff, but it's got great support for working with files, replacing content in files, merging files, file system options, copying directories, cleaning directories. Composer operations, you can run from within PHP. So we'll look at that later too. So you can do things like run composer update, really elegantly. Run composer validate to make sure those composer update changes that you did get your composer JSON file validates as you expect it to.

Working with version control. If you're using Git (UNKNOWN) support for SVN or HG mercurial. And then more general purpose stuff. So if there's not something built in, you can just call out to the to the command line using task exec or parallel exec, which if you're dealing with, you know, long running operations and things that can be paralyzed, this I found to be very useful and way nicer, way more elegant than dealing with the parallel command directly. So let's talk about the typical setup if you want to use Robo on your project. What you need to do is require consolidation Robo. Again, if you're on a Drupal project and requiring Drush, this is already there. And all you need to do is create a Robo File dot php in your repository root. This is what the most basic implementation of a Robo command in that Robo file can look like. So you've got a class, Robo file extends Robo tasks command. And then we have our command. Why is this command? This is a command because it is a public method.

Any method in your Robo file that is public is going to be exposed as a command. This, the method name becomes your command name and you'll see how that breaks down in a second. We have an argument for our command. With the documentation and the doc block and we happen to give this one an alias. And we'll see that on the next slide. And then we have an example of some nicely supported user feedback for the command line. So we're off to the races with this. And, this is how it all gets parsed out. So if we just run Robo, it's going to parse what it can find and expose our commands and we'll see it here on the last line. Hello, colon, MidCamp is our command. We have hi, is the alias and then we have our documentation that's parsed out here. From the dock block to our command. And if we run this command, This is what we get. Hello MidCamp 2023. Our argument that we passed in. This is as simple as it gets. You can add all kinds of commands to your Robo file. You can make them quite complicated.

You can have helper methods. You can make those protected or private then they won't show up as public commands. And you'll add, you'll start to love this. You'll add more and more. And before you know it, you'll have a robo file that's hundreds, many hundreds of lines long and it's got commands that don't really have anything to do with each other. Replacing those bash scripts you've got things for. Database backups next to things for test running your tests. So it starts to get a little overwhelming. If we're working in modern PHP, we're probably used to object oriented programming. We're used to separating concerns amongst classes. So that's what I wanted to do. I got to the point where this is unwieldy. I don't want to work in this way. I want to separate my commands just like I would my class classes in a module. So, auto loading to the rescue. We don't have to use that Robo file. We can, we can empty it out. Add an autoload directive for where we want things to live. And then in our repository, see here we've broken out a whole bunch of different commands with their own classes or groups of commands, let's say.

Because each of these classes, maybe they have multiple commands if they're related. So this was the next step on my journey with this. I was like, OK, I've gotten what I wanted. Everything makes sense. It's in different classes. But then I started to realize, I started duplicating this across projects, and some of these commands were specific to the project, but some of them were pretty similar, if not identical. Some of those database backups, some database downloads. So for example, with a single command, you could build your site locally, pull in the latest database dump from production, import that to your site. I wanted a similar operation across all of our projects at Chromatic. So we created a collection of commands that we call Usher. Usher is a public repository. The definition of Usher, to show or guide someone somewhere. It's also American singer, songwriter, businessman and dancer, Usher Raymond. So you can you can use Usher. You can also build your own version of Usher for shared commands or you could include, just include your commands and your individual projects.

And you can mix and match those two. You can have some commands that are shared and then some that are specific to an individual project. So you can require composer Usher. And those were the three approaches we've gotten. Robo file, commands within the project, separated (UNKNOWN) classes, and then we could share commands across projects. So now I'm going to go through a couple examples in a little more detail of how we have used Usher to help on some of our projects or many of our projects. So when chromatic projects we build every PR in a preview environment using Tugboat. Shout out to the Tugboat folks. Tugboat is also available on Drupal dot org. If you're not aware you can enable Tugboat on any contrib project. All you need to do is add a Tugboat config file and any merge request that's created, it will build a Tugboat environment for that. So you can then test the merge request changes, see that it built, see it in a live environment without pulling that in locally. It's really great.

So when we build an environment, we want to make sure, among other things, one of the things we want to make sure is that the Drupal configuration status matches what's in code. So this is a Drush command config status and it displays what's not in sync between the configuration in the database to the configuration that you've exported it to disk, or it'll tell you if it matches. And we wanted to run this on every PR so that before, while we're doing the review, before we actually do the review, before we get to approve and merge, we can have feedback that says yes, config matches, database to code. We feel confident in it and if it doesn't, we want to mark the PR as failed. So this is what a typical PR looks like that uses this. It just so happens, I don't think I did this intentionally. It is a PR that is updating Usher, but that's irrelevant. What is relevant is down here. And is all of our checks and the check that we're talking about today is the configuration check. So we'll also note that there is a check below this called Drupal Status Report, which is a very similar, we're doing a very similar thing.

We're using a Drush command to check the state of something that's going on in the Drupal database and that in that case is the status report. And in this case it's the config check. And that's important because we ended up with pretty similar bash scripts when this was originally written. So this bash script, which you don't need to read or know anything about what's going on, you should just be overwhelmed by the side of it. And it's actually not even as complicated as it should be, because if I'll point out that at the bottom, it actually is already running a Robo command. So this, everything above all this bash code is just the code that talks to the GitHub API. So the actual running of the Drush command and dealing with giving some good output to the command line and all that, that's already handled. But when we first implemented the integration with GitHub, it was written in Bash and then when I came along and said, I want to do a different version of this a second for those two commands, I ended up with a lot of very similar code.

And I wasn't comfortable enough with Bash to abstract it easily so that it could be reusable. So I wanted I didn't want to have all that duplicate code. I didn't want it to be in Bash. I wanted it to be more maintainable. This is gross. We're escaping strings. We're calling out to (UNKNOWN).

SPEAKER:
Have you ever tried to do JSON and bash before?

MARK DORISON:
I shudder just to think about it. So we're going to look at a few pieces of this that is broken up now. Here we'll go through a couple pieces. We see that we have a class for validate config commands. We're using a trait, a GitHub status trait. So all of that GitHub talk communication code that was in bash we put into a trait. So it could be shared across multiple commands. We'll see that trait in a second. We get to use things like constants and then we get to use all of this. Then we get to use things like PHP stand to do static analysis because it's in PHP instead of bash. So as we move on, this is our command. There's a lot of DocBlock up at the top to tell Robo what it should be expecting. But really, we can see all of it down here in the last three lines. We have a command which is going to get parsed out to validate colon Drupal dash config. It is a public method, so it is going to be called as the command. And then we have one argument which has a default and then we have an array of options, only one options which is essentially gets parsed by Robo and turned into a flag.

So on this command, if we run dash, dash, set, PR status, then it will flip from false to true and then we can change the, how it's going to be executed in the code. Any questions on this slide? This is maybe a good point to... Alright. Good. Cool. So, now we move on to our trait. Here's our GitHub status trait that's going to contain all of the code that was in that messy, disgusting looking bash screen. And this is some of the code in that trait. So we get to see, we have some, we've created some helper methods, set GitHub status pending. Set GitHub status access. So for example, when this first starts to run, this gets called, it sets the PR to say that it's pending. So we know just by looking at the PR, OK, this this has started, it's running, we should expect a response back. And then if it succeeds, we put some nice output on the command line. So if you're reviewing the logs, you can easily tell that. But then we actually call set GitHub status which we'll see in a second. And there's a similar method for failure and any other states.

Those are the only three states I believe. So now this and the next slide shows our... This is the meet. This is what we replaced all of that bash code with. And, all of this is the setup. So we have our set Github status. We have some arguments we're passing in. We're using some environment variables, grabbing things from the (UNKNOWN) environment. These are all things that we are going to use when we call GitHub so that you know what GitHub knows, what repo we're talking about, what PR we're trying to set the status for. So it's all just set up here. Setting the base URL. What URL we're going to call. But this is the best. This is the best part. We get to use Guzzle instead of cURL and everything that comes with it. This is when we're actually making the requests. We're taking all of those items that we set on the previous slide, the URL that we're calling out to, the access token. So GitHub knows that we are allowed to set this and then the body of the of the requests. And then we also have nice error handling if something goes wrong.

So the second example I'm going to show is one that I actually built while I was working on this talk. I was like, I forget what the exact catalyst of it was, but we use a lot of different environments when we're on most of our sites. So we use (UNKNOWN) for local development. Tugboat I mentioned we use on every project. We use GitHub actions to run all our CI checks that don't require a Drupal database. So stuff that does require a database and a live site we use Tugboat, but things like static analysis and CI we run at GitHub actions among other things like, if we need to do any custom deployment stuff. So this is a project that has a lot of GitHub actions on it. So but all of these environments... And also most of our projects, if they are hosted on places like Platform dot SH or somewhere else, they, these are all configured with YAML files, which is fantastic. And most of those contain the PHP version that we're running. That we want to run in. So when it comes time to make that change, we were ready to go from 8.0 to 8.1 or soon, from 8.1 to 8.2, someone has to update those YAML files and all those cases.

And there's also, typically it's set in the composer dot JSON summary as well. So, you know, somebody goes, makes a PR, you've got multiple change files with 8.0 to 8.1 and I get a review request to take a look at that. I go look at the PR and I say, hey, I got a bunch of files changed. They all show strings change from 8.0 to 8.1, but it's not so easy to remember. Like, how many files we actually need to change? Is there eight of them or five of them? It's it's easy to miss one. And all of a sudden, three weeks later, you're like, wait, why are we still running 8.0 in this environment? Or why didn't the README get updated that you need this version of PHP? So this to me was like, OK, this is a problem that we can solve with some tooling. So, this is the entire command. You don't need to read this. I'm going to zoom in, but this is the command I wrote and it's config update PHP version. So within this there's a fair bit of user feedback to the command line. So if we pull that out, this is really all that's left.

What I wanted to do was let people specify a number of files that were where we set the PHP version. And then replace those strings from the version that it was to the version that you want it to. So we have our command up here. It's a public method. So that is our command. We have one argument, the version string that we want to change it to. And then we have one option flag to skip composer update. And that's false by default. So we need to know what the current version is. So if we're going from 8.0 to 8.1, we need to know 8.0. And I didn't want people to have to specify that when they call the command. So we're going to store that somewhere. I'll show that in just a second. So we get the current version and then we get to config the paths where we want to check. So those are the two things that we need to, the two inputs that aren't provided by the user when they call the command and those look like this. We store those in in a YAML file. In a Robo dot YAML file. And this, is important for sharing across projects.

So if you weren't going to share this across projects or didn't want to, you could just put this in your command. I think probably the first version of the command that I wrote just had the (UNKNOWN) in an array and then we would loop through them. But I wanted this to be able to run on all of our projects. So it got moved to Usher, and then we show that set for whatever project you're on. Set the config paths, put whatever you want in here. We have a read me, composer JSON and all those YAML files that we were just referencing. So then we go back to the command. This is really it. This is all what it's doing. It's inside a loop for those config files. But all we're doing is using one of Robo's tasks, which we called out earlier in the talk, replacing file. Giving it the path, the current PHP version 8.1 and the version that we would like to update it to, 8.2. This was a little bit extra just for composer dot JSON. So if one of those files is Composer dot JSON and we changed the references to 8.1 and composer dot JSON, we want to run a composer update for the lock file to make sure that, because the hash is going to change and composer dot lock.

So I wanted that to be included as well. So as long as we haven't said skip composer update, if someone does run a width skip composer update will output that and move on to the next file. But if they haven't, we find the composer dot JSON file, we run again using Robo's composer update tasks with the dash dash lock option and then we run the validate command just to make sure that it likes what has been done. And this is what it looks like. We're ready to move from PHP 8.1 to 8.2. So we pass in 8.2. It detects that it's set to 8.1. Here to limit some of the output. I skipped composer update, but then it shows you all the files that have been changed and how many items have been replaced. It's nice. I like it. This is what it looks like. If it doesn't go so well you cant tell at 8.1. It says you're already on 8.1. And we get some clear, clear feedback for what's going on and what's gone wrong. Questions on any of that? So here we've used Robo to replace Bash Scripts. We've moved Robo commands into classes.

We've shared commands across projects and then we looked at two examples, Drupal config validation and PHP version changes. I'm sure you can imagine a lot of others and how that could be, how you could share that code across different ones. That's the meat of it. I will be around later for (UNKNOWN) time. If anyone wants to look at more of these Robo commands or questions, I'd be thrilled to to show them off and go through it with you. This is some unrelated promotion. If you don't know, we have a new podcast, the Drupal Seven End of Life podcast. It is not about the, not so much about the technical side of what you need to do to move off Drupal Seven. That is well documented. This is more about what life is going to be like for folks who are still on Drupal Seven at or after the end of life date and understanding sort of more of the less technical implications of having 400,000 sites still on Drupal Seven. And as we approach Life Day. It's been super interesting for me, I thought that we wouldn't have that much to talk about and we've had already great interviews with (UNKNOWN) and Tim Lennon and we already have a couple more lined up, so it's been great.

I have stickers for this too, if you're interested. And that's it. Does anybody have any questions?

SPEAKER:
OK. I just had a comment. There you were talking about, in the very beginning, like Drush has built on it or uses it like we've been using the BLT, BLT project. And that was my first introduction to the Robo commands as well. And yeah. (INAUDIBLE)

MARK DORISON:
Good question.

SPEAKER:
I was wondering if you ever do some analysis before writing the commands to think about what's the level of effort between making these automated commands versus just manually (INAUDIBLE). Is there like a level of effort where you would make a decision to go one way or the other?

MARK DORISON:
That's a great question that I don't have an easy answer for. I think... I kind of respond to pain and where things have become painful. I try to put myself in the shoes of all the different users of the project. So the person who is writing these commands is often not always the target user of the command. So I am much more typically by trade, more back end focus and clearly more DevOps focus. I often try to think about someone who is more front end focused and what their experience is like coming on to a project or using a project. So what are the things... And maybe they're even like not super Drupaly, for example. Like, what are the things? We have an (UNKNOWN). So there's a number of commands that we use on our projects that are related to... I kind of alluded to this at the beginning. ..Related to just setting up the project locally. So, running composer install, you know, downloading a latest version of the database, importing that database, running Drush deploy. Those are all things that if it was just for me, I didn't write those commands for me.

And so it's a number of commands that are then wrapped up in one (UNKNOWN) command. So they can run, they can clone a repo and then essentially run a single command and then have a running Drupal site when that command is done. I didn't write that command for me, I was like, this is fine for me. I know the six commands I need to run to get this up and running. I wrote it for the less Drupaly people who are working on the project so that they would feel confident. OK, I know enough of this, but I can be confident I can run a single command and I know I should have, if everything works as expected, I should have a running Drupal site with config imported and a database update hooks run and all of that stuff. So I try to think about those people when I'm running the commands. And I think there's a bit of it that's just like...

SPEAKER:
It's done.

MARK DORISON:
I think there's a bit of it that's, you just have to believe it a little bit. Like if you believe that it's going to save time and effort and expense, like, you know, deep down, I think. Like, whether it's worth it. Yeah. I don't know if that's like...

SPEAKER:
I think it's a very good perspective, especially to be thinking about not just the backend developer and whether it saves some time, but all the people who will be using it.

MARK DORISON:
Yeah. And also about building confidence and being like, OK, you know, like the change PHP config command, you know, that's not a complicated operation. We're talking about finding a replace or something like that. But maybe there's someone that's like, well... You know, if you did a find and replace on that project, the example project that I was showing that has that command, the string 8.1 appears a lot of other places beyond the, you know? Even in just like SAS files and things like that, it showed up in all kinds of strange place. So I'm like, OK, you could write some documentation that says the things that that command contains. It could say (UNKNOWN) in these files and all that stuff. But again, how many times do you go through that and somebody misses one and then the developer time spent going back and forth on the PR is like...

SPEAKER:
And if it's applicable to multiple projects that becomes....

MARK DORISON:
Yes. So and well, and that's actually something that is worth calling out. That project that drives a lot of the, some of these commands is a very complicated project and has a number of developers working on it. And so it has the... That project drives a lot of the worthwhile-ness of creating these moments. Well, it's worthwhile to create it for this project. Now, if we can, if it makes sense to build it for that project. But then we get to share it across all of our projects, even the projects that are really tiny and don't have a lot of hours allotted to them, then we can become way more efficient on those projects. We can spend our time on more valuable things that are delivering more value, more impact to our clients. And so that's where I think if you can approach it in that way, it goes beyond just the individual project, the individual implementation of where it's created. So that's what I get excited about.

SPEAKER:
It's (UNKNOWN) with a different opinion. That's not always bad.

MARK DORISON:
We have no shortage of opinions. Yeah. Any other question? Yeah.

SPEAKER:
What are some of the limitations of Robo? Because I was thinking like we had a (UNKNOWN) certain things for what you want to use it, but maybe you want a (UNKNOWN) unit or something else. And I imagine Robo fits its own kind of category when it comes to like automation. Where you might be like, I (UNKNOWN0 Robo, but maybe I should do something else just to have a performance increase. Or for some reason maybe, that side is not as developed.

MARK DORISON:
I think... I mean, the items you mentioned like (UNKNOWN) or what was the other thing? Yeah. I mean, so we call in, call out to all those things from Robo. So, we're triggering our, you know, PHP CS, PHP stand, PHP unit, all from Robo commands. Those are very simple robot commands. But it all runs through Robo. So people, because we have this kind of full suite, it can run similar commands for all of these things. And so for... This is kind of like the... Robo's like the orchestration piece of what we're calling out to. But to answer your original question, like when would it not make sense? I think.

SPEAKER:
I mentioned (INAUDIBLE), you know, there's multiple solutions you can have or testing. And I imagine for Robo there might be like multiple solutions that you can have in different levels of the stack. So at which point you like go all the way down to Robo and at which point there might be like another solution that is close by, but it's a higher level and you might switch to that next? If that has come across, 'cause you might have seen some of that while digging into it.

MARK DORISON:
Not... I don't know specifically if I have, but I think there's a balance and there's a value and consistency too. It's like, well, this is just the way we do it. So everyone knows, OK, this is where we look. This is where the commands are stored. This is the tool we use for this. And if there is another way to go about it that is significantly better in its approach and its results, absolutely, we should do that. But if it's relatively comparable, consistency wins in a way so that, you know, we know where all the bodies are buried and where to find things. So that's what works for us. But your mileage may vary. And just to to circle back to what I said at the beginning is, kind of relates. Like where Robo doesn't make sense is where, for me, is when PHP is not readily available. So I don't want to have to install PHP in an environment just to run Robo commands. I want to run Robo and use Robo because I'm working with PHP already. So that sometimes gets tricky when you are dealing with projects that have multiple stacks.

So if you have a PHP stack and a JavaScript stack separately within a project, sometimes that doesn't matter if the environments have both of those things, but sometimes if we're talking about containerized environments, they don't necessarily have all those things. So for me it's really gone the other way, but in our setups, sometimes you're in an environment where, we need to build a JavaScript, we need JavaScript to build a theme, for example, and we use a, like our front end team will build all of that with a JavaScript task runner. But then again, Robo sits above that and (UNKNOWN) when we build it into our (UNKNOWN) for like spinning up the entire site. That's part of what Robo does. It calls out to that piece. Now, that can be troublesome if you're like, well, we don't have JavaScript in this environment or a node, for example, in this environment it's a PHP. OK, what do we do? Do we install node in this environment so that we can build the theme here? And so sometimes occasionally we'll run into that.

But yeah. Anything else? Awesome. Yeah. Yeah.

SPEAKER:
Equivalent for JavaScript when it comes to...

MARK DORISON:
Sorry, what was it?

SPEAKER:
Is there an equivalent in JavaScript, like?

MARK DORISON:
Yeah. Rake I think. Or what was the one? Gulp? I think so, Gulp. Yeah. Yeah. There's no shortage, I think of... Yeah. It's the question of what your...

SPEAKER:
How many 100 words we spew out. (UNKNOWN) between Rake and Gulp.

MARK DORISON:
Of course. Of course. I think there... So I think the ecosystem is actually much more broad in the JavaScript community for tools like that. I don't see, I haven't seen like 20 alternatives to Robo for PHP test runners, for example. But you do see that in JavaScript. Yeah. Alright, that's it.