Drupal.org GitLab Acceleration Update

NEIL DRUMM:
Alright, So I'll go ahead and get started. I'm Neil Drumm. I work for the Drupal Association on the engineering team there. And so the Drupal Association we're the people who keep drupal.org running off Drupal.org, including the subsites like events.drupal.org, Drupal CI and git.drupalcode.org of course. And yeah, we're only four people on staff, plus tag one for infrastructure support for the engineering team. So just kind of recap from the start. Why GitLab? So we want to improve the developer tools on Drupal.org, making it more smooth for people to contribute code back to through Drupal. Anything that goes to the repository. And, you know, a few years ago we migrated git.drupalcode.org to GitLab. But we're not using all of its capabilities right now. And GitLab, as a company can do more than our small team as far as providing functionality around Git. So it's not just GitLab we're doing, we improved the whole contribution process. So that includes account creation, making it easier to sign up for an account on Drupal.org and that includes things like social signance.

So if you want to sign up with your GitHub account or Google account or GitLab.com account. Projects and releases, those aren't changing a whole lot. Those are staying on Drupal.org projects. They have some metadata that's not in the repository and project balancing on Drupal.org. And releases, those Drupal core specs packaging and as part of this we're also able to simplify Drupal.org a bit. And we'll get into that a bit more later, but it'll help us upgrade Drupal.org in the future if there's less stuff and more simple stuff to upgrade. GitLab CI and issues, of course, are being migrated. That's kind of the end goal. So account creation. Let's see. Account creation. Just to be in line with all my notes from when I wrote these. Yeah, basically what I said earlier, we had to kind of essentially custom solution before, a couple other places used bakery module. I think Acquia used to use bakery module for something on Acquia.com, sort of switching to key cloak for single sign in and social sign in, and that'll also provide federated login.

So a campsite like if Mid Camp wanted logging into the Midcamp.org site with your Drupal.org account, that should be possible in the future. So that causes some other changes. Multiple email addresses. Those used to be on your account on Drupal.org, now git.drupalcode.org manages those. So good simplification there and notifications are enabled via GitLab. And you'll see the login page will look a little bit different in the future. Projects and releases. Those aren't changing a whole lot. You know, we have project browsing on Drupal.org and we want to support project browsing within Drupal itself and that needs metadata that isn't in Git and you know all these pages that aren't in Git. And Drupal core, it expects packaging of releases updates to the info YAML file and packages at Drupal.org, that's part of what you use when you install the Drupal site with composer that handles some, making Drupal dependencies work for composer. And those notifications, those are under settings, under preferences on git.drupalcode.org.

So they're a little bit hidden but yeah, you can set up notifications and this is also where SSH key management and email management went. And as part of simplifying Drupal.org projects pages changed a little bit. There used to be on the top right, the maintainers used to be the, how many commits they made, which it's a little unnecessary focusing on code contributions and something Drupal.org just didn't need to do as far as keeping track of all the commits. So we no longer maintain a database of how many commits someone's made. GitLab has a few visualizations of commits over time, which are better than what we could provide. And let's see. Documentation that used to link off to just the guides on Drupal.org, now you can put any URL in, so if you want to use GitLab pages in the future, that's a place you can put documentation. And in the future when issues migrate, that whole issue account block that will have to, something will have to change with that because we won't be able to count issues anymore because we won't have a local database to get those numbers quickly.

Profile pages again. There used to be a count of commits at the very bottom of your profile page. That's replaced with a count of issue credits, which we can easily provide, and it's little bit more inclusive of other types of contribution, not just code contribution. An example of something GitLab provides that we would have taken more effort for us to provide in the past if we wanted to build it or had time to build it is code search. So this still took a bit of effort for us to build out, basically setting up, I think Elasticsearch server, Elasticsearch server for GitLab to use, but it handled all of the indexing and everything. There's one big trick to it. We only index the project group. There's also issue forks and sandboxes, which we don't index. So that means you kind of have to get to the project group before you search, and then code search will be there. And big thing that's in progress right now is replacing Drupal CI. So, you know, the aim is to get GitLab CI to be feature-complete with Drupal CI.

So testing both core and contrib projects and the current Drupal 10, 11 core and Legacy Drupal 7 that's still maintained if we need core and contrib tested for those. And Drupal CI it's everything from the container stack. We have to custom build some containers to get extra debugging output and things like that. Actually the big thing is Drupal supports a really wide variety of PHP versions and database versions and stuff that other people aren't maintaining containers for. Like Drupal 7 has some amount of PHP 5 something compatibility and no one else cares about those containers. So we have to build them. And we'll keep building them with GitLab CI. It's all containers, just changing how we orchestrate them. Assembling a Drupal code base of contrib testing that's assembled in roughly the same way. Linting, PHP lint, CSS lint and actually running the tests and reporting the results. So the big difference with GitLab CI is, you know, with Drupal we had patterns established. We could code those into Drupal CI.

We know what testing a Drupal project roughly looks like. But with GitLab CI of course it's, you know, you can put any projects on GitLab, there's no standard way of testing them. So GitLab CI is going to be a bit decentralized. And the other kind of big win is GitLab CI is going to handle all the orchestration. Right now we have a Jenkins server, you might have seen dispatcher.drupalci.org I think is the domain. You really shouldn't be running a public Jenkins server like that and it's not really designed to, you know Jenkins CI system is OK for this, but we're kind of pushing the limits of what Jenkins can do PHP is here to have GitLab do all of that work situation running all the mechanics of running the tests. And, yeah, as a maintainer, since GitLab is more decentralized, you will need to set up a GitLab CI YAML file in your repository and that will have all the configuration replacing the UI on Drupal.org as far as like what to run on commit, what to run on weekly, what to run, the scheduling of all the tasks.

And, setting up a GitLab CI YAML file can be work for every maintainer who wants to keep their testing running. So we want to make that transition as easy as possible. The good news is we have templates that basically you can include portions of that YAML from somewhere else from these templates. So those get you started with testing a Drupal project. Right now, those templates are in a good spot to start I'd say, maybe better quality and we're ready to have people, project maintainers, request GitLab CI and test out those templates, see how they work, provide feedback to us and help improve those templates. So the slides are linked from the Mid Camp site, so you don't have to type that all out. But we have that documented and in the next few weeks we'll enable CI for all projects, probably.

SPEAKER:
This is great news.

NEIL DRUMM:
Thanks. And of course, GitLab CI has no idea what patch file is. So when a project switches to GitLab CI, they're also saying you have to use virtual classes to contribute. So no more uploading patch files, no more inter diffs and all the mechanics around that. So, different set of tools to learn with merge requests. Yeah.

SPEAKER:
Is there any sort of reply or feedback for contributor coming to a project (INAUDIBLE to show them that, don't submit patches or is it up for the maintainers to sort of give them that feedback?

NEIL DRUMM:
So, I'll repeat the question. Is there any feedback on the issue page about issue patches are no longer accepted for projects using GitLab CI? Not currently, but we'll certainly need to add that once GitLab CI gains traction. And yeah, we do, as the maintainer you can use Drupal CI and GitLab CI in parallel for a little bit until you get get GitLab CI up and running testing everything correctly. We do ask that you don't run both indefinitely because we do pay for the compute and everything to actually run those tests. So, yeah, we'll have that in the future.

SPEAKER:
What's your assessment for the state of the documentation for someone who's interested in GitLab CI (INAUDIBLE).

NEIL DRUMM:
So I didn't write the documentation, my coworker finished writing it two days ago. So it looks pretty thorough but, you know, not a lot of people have used it and it was finished two days ago. So I'm sure it needs work.

SPEAKER:
It's gotten attention. That's goodness.

NEIL DRUMM:
Yeah.

SPEAKER:
Yeah. You've had a. Which is always.

NEIL DRUMM:
So once a project is moved off of Drupal CI and testing is working for them with all GitLab, then we can now look at issues. So the thing that's different about the Drupal project from the other open source projects is we do things like contribution days, have contribution areas at conferences and sprints. So it's a lot more common in the Drupal community to have changes to the code handed off from person to person. Not the same, you know, I might start a merge request and say, I don't know how this should be tested and just leave it for someone else to pick it up and add the testing code and that's not the way you see GitLab or GitHub usually used. It's usually one person and one change. So yeah, I ran the stats a few days ago. On average 5.6 people are credited for every Drupal core contribution and you know, they're not all coding. But you know, there's a lot of non code contributions around the receive credit but it's expected that the code changes are handed off from person to person.

So we have to figure out how to keep that collaborative sense and that's why we have the issue forks and merge requests set up the way we do. And credit, that's also something that's become really important in the Drupal ecosystem that is not seen in the same way in other open source projects. And before getting into that issue credit too much, I want to remind people it's not just issues that are on your plate. I consider credit to be anything that shows up on your user profile, anything that shows up on your organization profile if you have an organization. So, we've added contributor roles, stuff like ongoing roles in the community and documentation that you've maintained that's linked from your user profile page. And what else do I have on here? Yeah. Organizations, if they're supporting a project as in sponsor maintainer, that's been in case studies. But for GitLab, the main thing it's going to be affecting is issue credit. So GitLab, they don't have anything like issue credit built in.

And GitLab as product is not like Drupal, where we can just add something to a page. So, we'll be rebuilding it on Drupal and pulling in data from git.drupalcode.org. And, the only interface we have to put stuff on the screen in GitLab is places like posting a comment. So, there'll be prompts in the GitLab UI for, Hey, this issue was closed out. Remember to update your, if you are contributor to this issue, remember to, say if it was on behalf of organization or as a volunteer and if you're a maintainer, go say which contributor has made useful contributions. And you of course can do this throughout the issue lifecycle. You don't have to wait until the end but it'll be a separate system over on Drupal.org. We have a bit of UI work to do, but you know, it's all filtered data and stuff Drupal can handle as far as actually building that.

SPEAKER:
(INAUDIBLE)

NEIL DRUMM:
So the question was if it's all manual or through the CUI or if you can, (INAUDIBLE) Well yeah, the commit message, we can pull that in the commit messages that... I have a feeling this might have to change the formatting. The standard Git commit message to credit someone is commit trailer like the signed off by colon, email address reviewed by colon email address, kind of like headers at the bottom of the commit message. That's more likely to be fit into how GitLab will make commit messages automatically. The whole listing usernames at the front, that's not something I've seen other projects do. And it's a little harder to parse because usernames you git.drupalcode.org is actually technically could be different from your drupal.org username and people change usernames. But yeah, it will be some amount of UI on a separate site to remember still. There's also stuff throughout Drupal.org that depends on issues such as the change records. So anything that's referencing issues on Drupal.org.

We have this nice kind of color coding that we can't do that anymore because we will have issues in the local Drupal database. So to unwind all of that integration and replace it with GitLab integration. So you might have noticed Change records have changed slightly in the last few months. That's where we're preparing for the unlocking. Yep.

SPEAKER:
(INAUDIBLE)

NEIL DRUMM:
Yeah. So case by case, some things, those references to issues on Drupal.org, they'll just go away and we're not replacing the functionality. Places like change records, it's important to keep that functionality. So we have to build a new integration with GitLab, which will be more code than integrating with Drupal because Drupal has any reference module built in and stuff like making references to other content on the site, it's natural in Drupal. But that depends on having the entity there in the database so you can query it. And GitLab, you build API integrations. So stuff like, if we want to do that color coding, we can make a request on the front end in JavaScript to check GitLab API, Hey, is this open or closed? What color should we make it? So yeah, the JavaScript code is you know, it's not that much more than entity reference and whoever was doing this in PHP, but it's a reimplementation potentially a little bit more, more code or code base. So kind of the theme is like, this isn't saving work or complexity on Drupal.org not having issues.

Yeah, that's a little less complexity there. But, you know, about half of the functionality on issues was the contribution credit. Like, that's not moving since GitLab doesn't have it. And then the actual migration of issues once solved, the, you know, once all of the things kind of coupling issues to Drupal are uncoupled, then we can actually start migrating issues. And the way we'll do that is project by project. So it won't be a we can't migrate everything all at once. And you know, we aim to preserve as much as practical of the metadata. GitLab keeps track of metadata in a little bit different way. They don't have the, you know, the eight fields or whatever we have for status and priority, but they do have labels that can replace that. And so we got a little bit of a head start in the code. For that, we have a draft that's, you know, of the code that can migrate issues. But of course they'll need a lot of review and we have all of those links to Drupal.org like, issues, those references and Drupal CI.

So, yeah, this is where we are. And also, the little Drupal icons on these. You know, we're also, Drupal.org site, Drupal seven site so we're looking to ways to simplify, have less to migrate and once we move to Drupal 10, stuff like the issue credit, we're building that UI completely on Drupal 10. We're not going to build it on seven and then migrate it. And, yeah, the meta issue to follow along is there on the screen. there's a Drupal slack channel. Best way to help is to help test things as they become available, especially merge requests and especially GitLab CI. Now that we have that documented and we're ready for a few more project maintainers to give that a try. And also, there's a couple people, probably only one person working on making the testing work for Drupal core. So we'll need that as well. And yeah, look through these slides and the feedback form are on the Mid Camp website. Any questions? Yeah.

SPEAKER:
So GitLab of course includes, are that all the templates are intended to be used?

NEIL DRUMM:
Yes. Yeah. So yeah. GitLab CI YAML, it's a whole syntax. And I don't personally write a lot of GitLab CI files, but it's a programming language crammed into a YAML file and it has (UNKNOWN).

SPEAKER:
Great. That was my big (INAUDIBLE) and everybody just includes that template that's hosted by the different people or whatever and they don't need to customize it or (INAUDIBLE). Awesome.

NEIL DRUMM:
So yeah, in the future, when people come up with good automations like the Project Update Bot Run Richter and all the other tools in that suite. Yeah, they can either in our default templates, we could add a build step, run, was it the project update or? Yeah, run project updater. And either, hopefully in the template say like, by default, run it weekly or something like that.

SPEAKER:
So are there going to be any new restrictions (INAUDIBLE) or?

NEIL DRUMM:
No. So, yeah, we don't... The question was, are there restrictions on the types of containers. And yeah, we're not able to put restrictions in for that. So we will be monitoring how many CI minutes people run their tests for because we don't want to pay for abuse of the system or just over testing. Like if you're testing every supported version of PHP with every database, multiply that by every supported version of Drupal core, you could easily get to, you know, ten, 20, 30 tests running on commit. And you don't need to be running 30 minutes or an hour of testing for every commit or. So, yeah, we will have to limit how many minutes the projects run for because we do the compute costs money and we're outside of all the free tiers of GitLab.com. They'll give you a little bit of testing for free but not the amount we need.

SPEAKER:
So, like these templates are they're going to have the ability to save (INAUDIBLE)?

NEIL DRUMM:
Yeah. This is what the documentation looks like. I think that's, So, it has scheduling and... I can't read any of this since it's up on the screen. But, this sort of, this is where we are with the documentation so. Far. Any other questions?

SPEAKER:
Can you talk a little bit about, I'm sure you've had discussions about timing. Projects that are potentially very old and don't really have people maintaining them. But people might be using those projects and have composer files with patches. I guess, as we go on, eight to nine and nine to ten, things will be sort of deprecated by default, (INAUDIBLE). Have you had much discussion about timing and about when will there be a hard cutoff where the issues system will not work anymore, where patches won't be available? (INAUDIBLE)

NEIL DRUMM:
So, yeah, around timing, everything we do will be project by project. Right now it's an opt in process for GitLab CI. And be like that for some number of months likely. You know, even when we enable the ability to use Drupal or GitLab CI for all projects still requires that manual step of adding GitLab CI YAML file. And yeah, at some point six months or a year from now, however, the timing works out, we'll wind down Drupal CI and exit its technical debt. We can't be maintaining two CI systems.

SPEAKER:
So would you still be having those patches available? (INAUDIBLE)

NEIL DRUMM:
Everything in the SUQ will be migrated. So yeah, patches will become attachments to the GitLab issue just like any other files on the issues.

SPEAKER:
(INAUDIBLE)

NEIL DRUMM:
Yeah. Although probably redirects and everything. Like we aim to keep the transition as smooth as possible. So any images uploaded to the issue, those will either be uploaded to the GitLab issue or we could link them over and keep them hosted in our static file server. But yeah, they'll be available so you'll know every image, every patch that's been uploaded previously. And a quick guide since you mentioned patching Drupal installs, you can get a patch for a merch request, dot patch or dot (UNKNOWN) URL on GitLab on the merge request. But don't deploy that. Don't reference that directly because any merge request, any pull requests on GitHub, it could change in the future. And you probably want to know what you're deploying. So you know, make a local copy of the patch, apply that with composer patches or if you have a bunch of stuff going on in your code base, maybe make a private fork like you would do on any other project you would be deploying. That's all? You had another question?

SPEAKER:
(INAUDIBLE)

NEIL DRUMM:
Yeah. As far as automating, enabling GitLab CI and actually using GitLab CI, I could imagine potentially, these GitLab CI YAML files for each project, they'll follow a pattern pretty well, and we could potentially generate a good place to start and open a merge request for each project and say, Hey, this looks like a good place to start. As a general rule, we don't make automatic commits to your repository. We've stayed away from that.

SPEAKER:
(INAUDIBLE)

NEIL DRUMM:
Yeah. GitLab does have project templates for new projects, so that is something that could be set up.

SPEAKER:
(UNKNOWN) And in addition to that, there's an option to submit to the GitLab itself, the option to have default template. (INAUDIBLE). But it takes work and takes time.

NEIL DRUMM:
Yeah. I'm not sure I can summarize all that for the recording, but yeah. One of the learning experiences around using GitLab has been, you know, our old system is maintenance free. Nothing changed. So there was no real maintenance burden. Sometimes, you know, we would look at it like, how is this still running? There's like a python SSH statement somehow that was still working. But don't touch it. It keeps working. GitLab, they're on a release schedule. They put in a bunch of new features every month. So they have security releases every once or twice a month. So, you know, we're in the same position everyone else is with updating Drupal core potentially every month. You want to update it pretty quickly and we're about around the beginning of the month, we know someday there sometime there will be a GitLab security release. So do those updates promptly and this is the disk use for that past six months. Something changed around January 1st, and now it's becoming a problem. So we need to figure out where to get 40% of the disk back or the other solution would be buy more disks or provision more disks on AWS.

But things change every month and this one, we kind of let it ride for what? January was four months ago, three months ago but now it's becoming a problem. So that's that's part of, we're not just moving things over to GitLab, we're also doing production support being site reliability engineers for this product. Yeah.

SPEAKER:
So it just checked. And not only when you're setting up a new project, but also an existing project GitLab provides buttons to add or renew, add a license, add a Changelog and set up CI CD. It looks as though if you use that button, you get the standard GitLab YAML file. So what sort of the UI is there and what you have to do is figure out how to get the Drupal specific CI configuration so that you get that report. So yeah, the question or comment was, yes, GitLab has all of these kind of, on GitLab they have a business interest in using GitLab CI because GitLab.com you know, you'll pay for it. And their business model is to become, I think full stack like ideation to deployment system for code. So they want, so GitLab CI there's all sorts of on roads to enabling GitLab CI but there's a few, there's some options with that we can keep finding more ways to put the Drupal templates front and center like have that, I believe when you enable GitLab CI for a project through the GitLab UI, it gives you a choice of templates, and I think we can add stuff to that list.

But yeah, there's places we can make that more kind of the defaults or available options. I would love to see that for new projects. I think there's still this value for existing projects, the great work we're doing. You know, not committing code (INAUDIBLE) but promoting that this work is available in places that (INAUDIBLE). The other example I would give (INAUDIBLE) a lot of people don't know that that exists. And, you know, (INAUDIBLE) somewhere in the project page or maintainers that you can (INAUDIBLE). So I think there's a bit of like, it's kind of a marketing piece to the technology work that you all are doing, and it's really great to see more people made aware of it. Yeah. I definitely have seen in GitLab, you can have customised new project templates, so we could make a specific Drupal module template. We never want to, you know, we also want that to be an option because you might be bringing your code base from somewhere else, importing, Git repository, so you should always have the option to start with a blank repository, but we should also have the option to start a what most Drupal projects look like, themes, modules.

(INAUDIBLE)

NEIL DRUMM:
Yeah. And we can keep adding stuff like more advertisements of features to Drupal.org. Within the GitLab UI options are a little limited to, you know we can't add arbitrary things to this new page other than or merge request page other than comments. But yeah, part of the collaboration will be a comment made by a bot on the issue, like here's the fork for this to contribute code, just a little bit of information on how to get started. Yes, go ahead and commit stuff to this. It's fine. And here's the commands to set it up in the, set up your get copy the way you would like. Or use the GitLab Web ID there. They're also pushing that. I think I saw the latest release of GitLab has like more autocomplete stuff in within the GitLab web ID itself. They are on the whole AI bandwagon. See how it does.

SPEAKER:
(INAUDIBLE_

NEIL DRUMM:
Right, any other questions? Thanks.