Drupal migrations and upgrades

SPEAKER:
So, I can start maybe. I just find migrations to be really difficult and confusing and I’ve already done two migrations. One was a six to seven a long time ago and one was a seven to eight. It started about a year ago and finished about six months ago. And then I’m just starting one now that's migrating just blogs from a WordPress site to a Drupal eight site and they're actually started to use the feeds module instead of migrate because I thought it might be easier with an interface and stuff. So, yeah. It turned out to be easier So far, so far, so far it is in terms of the blog is pretty simple in a blog post, it's just got a title, it's got a body which has a whole bunch of embedded images and in the body it has some tags so, they become taxonomy. So, those things have migrated or I shouldn't call it migrate but got pulled in with feeds, OK. The title the body, the two types of tags, what's remaining to do is this one featured image. So, I’ve remained to do that and what I really get confused about is media like how do you take an image file over here like over WordPress, over Drupal?

Well, I know how to do it in Drupal seven because I did it but it seems complicated. How do you take that an image over here and make it media override the Drupal eight site? And then hook that media into the field that's using it on the content type that you migrated, you know. So, I like I said I did that for Drupal seven site, so, I know it's possible but I have no idea how you would do with a WordPress site. And I know the answer to that is probably not simple, so. Just arrived and missed the first half of your question. What is hard to do in a WordPress site? So, I just find migrations hard, period. And then I’ve only done two migrations, one is from a six to a seven and one is more recent from a seven to eight. So, I’ve done but I just find them difficult, I’ve just started a migration from WordPress to Drupal where all I have to migrate a blog post. So, I actually started doing it with feeds in Drupal eight thinking that would be somehow easier because it's a nice interface and feeds and all that stuff and I’m getting here.

I would definitely say do migration. I thought, let me google this, I thought that there was a WordPress plugin for migrate. There is. I started to use that and it says to export the WordPress stuff is xml but when you do that the xml for blog posts is weird, it doesn't actually contain the categories in it or anything like that in. So, it's just weird xml, so, I kind of gave up and export it as csv instead. And then started to use feeds but yeah, it'd be better to use migrate, I guess. I mean you could go right against the database just, you know, make a (INAUDIBLE) migration. Yeah, that's maybe part of the confusion. Not so much when you're going from Drupal to Drupal but if you go in from anything else to Drupal, what are the possible methods? And which one would you, should you try first like should you try to export as a csv and use that for the migration? Should you try to hook into the database and use that for the migration or what? Well, I think I’m about to give you any answer you were about to drag that depends.

Yeah. It all depends on what your source data is and how easy it is to get stuff out of it and like you said, how easy the databases to parse up on your own or, you know, your data source. Like I said, hooking into a MySQL databases and isn't difficult, again almost positive there is a MySQL plugin already, I think I may have had to write a custom source for that at one point but I don't know that you do anymore. (INAUDIBLE) source, yeah, no, it just kind of supports it natively. What's the name of the plugin? Is custom sql migrate source plugin. Oh, wow, this one now actually lets you do, you can specify a query right in your in your source config. I'm sorry, I was talking at the same time I did. And I didn't stop. Go ahead please. I was going to mention like, you mentioned that going from six to seven, and from seven to eight, I just want to acknowledge that the system is completely different, like Drupal seven was PHP based for the most part. In Drupal eight we have, you know, it's under the hood, everything is converted to PHP code, I guess.

Initially, it's (INAUDIBLE) base and the architecture and the options are a little bit different. So, I would, like I have done probably many migrations in eight plus, like eight or nine, and not as many in seven. And it's a completely different mindset sometimes. So, it kind of, it's relatable. One thing that I would suggest is that we all maybe introduce ourselves and share, you know, a problem or an idea or a challenge or something a little late to take from this conversation. Based on the schedule, we only have about 30 more minutes. So, if we see some common patterns, we can then address them in batches. Yeah, that works for me. There was a long pause, I guess I can introduce first. So, I'm jack Franks. I'm mostly back in Drupal eight developer and boy I love migrate. Drupal eight migrate is one of my favourite core functions. I don't know if it's going to point back to your own talks. But I've got a bunch of talks on it from mid camps before that might help you out. If you're really tried to put custom stuff together for migrate not just some Drupal version to some other Drupal version.

Migrate is so much fun, I don't have any particular problem. I just like talking about it. I'd love to share what I did on that seven day eight migration and see if that was the right way to do it at some point. I'm Benji Fisher, one of the maintainers migration subsystem. My name is Marty INAUDIBLE). I've been doing migrations for four or five years, and I've been enjoying the challenge of learning and implementing the API along the way. Also, if you look in the chat, someone's posted a link to a place where we can share notes. So, Jack, if you'd like to post in the link to some of your presentations, if that's possible. (INAUDIBLE), you could always put those in the link for the 31 days of migration series. I'm Joe. I don't I have some experience with migrations, I don't have any particular questions and was more just kind of curious to come and see what other people were doing with migrations and learn something along the way. Sort of in the same boat. I'm Noelle and I work in a (INAUDIBLE) triple cloaks where I work.

And we have about a handful of triple seven sites that we need to migrate. Obviously, we're a little bit behind in an area. And we're also any new sites that we pray going forward, of course, on Drupal nine, but I just kind of wanted to join this group and or join this session and hopefully learn a little something, get some tips, etc. We're going from seven, nine is (INAUDIBLE). My name is Brian. And I'm also interested in seven to nine migration, which we're going to be doing later on this year and next year. I really don't have experience with migration. We did a project last year that we migrated from a dotnet website into Drupal eight site and I was involved with, you know, planning and mapping out fields and things but the whole mechanics and coding the actual migration scripts, I know somebody who knows more about that stuff did. Terry and I have done an incomplete, I have to say it was incomplete because I never did get quite finished six, two. I think it was 8.6 migration about a year ago.

And one of the issues was exactly what Rick was talking about. And that is how to handle, how to handle bringing in assets into media in eight slash nine. So, I don't have to revisit that particular migration again and because the Drupal six site is still up. I never did get it 100% going so, it's still there, it's I got to finish it obviously. But in the process, I also managed to accidentally blow away the database that I had migrated to. So, I’m really looking at starting all over and going into 9.1 and perhaps curious also about, I mean, at this point now the site would be on php seven four as soon as I can't get it into php eight I would do that. And just wondering if that makes any difference in terms of how you would handle any of these migration issues. So, for anyone just getting started, I recommend Mauricio series of blog posts which he pasted a link into the document. On the question of getting things into media, the general process is you first do a migration that creates file entities.

One way of saying that is that you're populating the file manager table and then you have the second migration with the same source that creates media entities corresponding to the file entities. So, I’ve done that many times. I obviously have a bias towards the migrate api, I so, keep that in mind when I give the advice that use feeds only if it's easy, if it's hard use the migrate api. One common that I would like to add to that is sometimes I see people especially in slack when where the community offers support like jumping too fast into writing the migration and I would encourage everyone to take a step back and try to understand both your source and your destination very well. And in particular something that is going to help you a lot when you face challenges is like understanding the relationship among different entities in Drupal like, OK, for example if I have media field in a node, I need to be able to identify where all the entities that are related to that. So, I have a note entity, I have a media entity, I have a file entity.

So, understanding I have three different entities that I need to create in order for this to like glue together, that is going to give me a clue as to how to approach the process. For the most part, and this is the, you know, recommendation, there are always exceptions. You're going to have one migration per entity. So, one for the file, one for the media entity and one for the note and then you should be able to connect them. And that is something that I see often like if you're in a node you have taxonomy term is anything that is an entity reference field or an entity reference revision field like paragraph that is going to give you a clue that is going to be very likely a different migration. That and you need to be able to establish that relationship about among the different entities. That being said, also like understanding your source is key and from then on, it's just like understanding your tools. So, the one of the best things is like there is a page in the documentation about lists of process plugins.

If that's weird or you're not familiar with that concept, I recommend, you know, researching about that but it's just like that also that you will have at your disposal to work on your migration. So, it's like understand in general Drupal site building concepts, understand how the site is modelled in your source and your destination and then understand the tools that are available, I have pasted some links specifically about, you know, how to get started, two series, two articles and two podcasts in the shared document. One of the things that I’ve done and it's been super helpful for me, work especially working on with a team is as part of like trying to understand the source site is a lot of work in spreadsheets initially and we'll do things like create a spreadsheet that has a list of all of the modules on the Drupal seven site. And start to figure out like, is there, is there a Drupal eight or Drupal nine version of this module? Yes, or no? Is there a Drupal eight version, but it is a different module?

So, like the community has coalesced around a different contrib module in Drupal eight than in Drupal seven, and kind of getting a picture of that. And then we also frequently will make a spreadsheet that contains a list of like all of our content types, basically all of our like entities on the Drupal seven site, and all of the fields for each one. And basically start creating notes of like, this is what we need to do to migrate this particular field. And we often find that like some of them are, a lot of times, like migrating a particular content type. It's really easy, except for like one field that you have, you know, it's like, oh, we're trying to change the name of it, or this field type doesn't exist anymore. Just whatever weird things like that. And being able to look at that document where we have kept notes as a team about like, essentially, like, these are the parts we already know how to move. And these are the parts that we don't know how, and we need to figure out That make sense, when we're doing something similar trying to get ready for migrations.

And it goes back to understanding the source and the destination. And if I'm not mistaken, doesn't, does the migration module or my thinking of something else, actually give you suggestions? You know, if a module doesn't exist in seven, what the community recommends or is gravitating toward? There's a tool somewhere that I found, but it's been a minute since I've looked at those notes. The migrate upgrade module, I think that's (INAUDIBLE) Migrate upgrade, that's it. I think that like the module itself, I don't know if it gives suggestions. But that when you mentioned that, (INAUDIBLE) them (INAUDIBLE) module that you run through the UI, and you get a report of the availability of modules in Drupal eight or nine. And then, and then what are optional, options for things that are not available? That is migrate, excuse me, that is (INAUDIBLE)? Yeah, I think that is it, actually. Yeah, that's what I was thinking of (INAUDIBLE). And that is one thing, it also tells you what to (INAUDIBLE) in Drupal eight, nine, that wasn't seven.

Like these, for example. These are always such interesting sessions, because the talk is so often about these, you know, Drupal version updates. But by far, that is the least common thing that I use migrate for. I use it for everything and only rarely have to update sites. Lucky I recently came up with an interesting use, where we're building a new site, it's not migrating from anything. But while we're building it, the content editors are of course, copying and pasting links. So, they start with, you know, [email protected] slash about us. So, you can use the migrate API to search through all of body fields and all of your nodes and replace links that start with mysite.hosting.com slash and just snip that part off. When or thing that I would recommend for anyone, either learning in preparation for an upcoming project or already doing the work is for, you know, anything on this East Coast region, right. And for anything that you want to learn, try to look for an example and try to make the example work for you.

Like in the case of the migration, you know, look for an example online and try to get it working locally. And it's easy, especially with the migrate API and how (INAUDIBLE) to make mistake, one extra space one missing space can break your whole migration. So, starting from a point where you know that it is supposed to work and you get that working, you know, you have made progress already. And from there, you can either tweaking the migration to fulfil your specific needs or breaking the migration on purpose to get yourself familiarised with the errors so that when they happen in the future, you don't, you don't get caught, you know, like, by surprise, you are already familiar with the type of errors and you know what might be the cause of, the source of them. So, yeah, that's something that I found myself very useful to do. And I have posted in the shared document, as links to different examples that I have written myself. And some of them include accompanying articles with the explanation.

But certainly, for example, in there you, there is one example that is about migrating media. So, you can start with that if you want to learn how to make the media. It goes from, you know, create the files, create the entities, create the notes and connect everything together. But in general, I would advise to look for working examples and then start from there. And Rick pasted something into the document, using the migrate upgrade module, and then running migrations by Drush. Yeah, that's a very common workflow. One thing you didn't mention is that once you've exported them as configuration, you have the option of editing them, and tweaking them to your heart's content. And then, you know, there are a whole lot of them, it really helps to have a simple bash script that runs them in the right order. You know, sometimes the order that's detected isn't quite right. And it's an iterative process. So, you know, be prepared to wipe out your database and start again from scratch, every time you do a tweak, and work on one problematic migration at a time.

Yep, I did that a lot. So, migrations in general, are all in (INAUDIBLE) files. Usually, you don't need anything more than that, right? I often write a custom process by thing or two. And, then I often end up contributing back to the migrate plus module. OK. I, when working on some projects, I will also have had the need to write source plugins from time to time to, you know, prepare the data, to sometimes clean up the data as part of the migration process in the source plugin. Something that I have never done is writing a destination plugin or done by just like curiosity. But like I (INAUDIBLE), that has never been the case. But that being said, something that I really enjoy about the migrate API setting is Rupali is that because it is (INAUDIBLE) based, you don't need to be a programmer to write migrations. And you can go a really long way without having to resort to writing code. Like one of the examples that I pasted there is like a full day workshop, where you never touch PHP at all.

So, it is possible to do a lot. I mean, of course, if you know how to programme, you will have more tools at your disposal. But even if you don't, it is possible to do a lot. So I heard Mauricio say, one migration per entity. And then I heard Joe talk about fields. So, my question is, in my Drupal seven site, my media entities have a field for image, a field for caption, and a field for photo credit. If I set all those fields up in the new site, and I do one migration that brings in all three at once? Pulling in your data fields, and you need like your media fields, your picture fields, anything that like if you're if you have an image field on there, that will often use a file entity and reference it. Those you need to split up. Well, you don't need to split up, you can do them all in one go. But it's way more difficult to manage, it's way more difficult to roll back. So, I would do all of your regular data fields in one migration, and then any of your sort of dependency based fields like files, in a separate relation migration with a lookup plugin, Or... OK.

Or first do the file migration and then do one migration that does all of the fields on the media entity. So, one of those will be the reference, the file field. And the others will be your whatever text fields you had or whatever. There you go, that was clearer than how I described it. Yes. I would look for a screenshot that explains some of these, like the relationship between entities and how you can migrate them, (INAUDIBLE) to find it now. OK, thanks. Well, or challenges or successes stories have people had with the latest API or migration products in general? Are there, there's some questions in chat? David, do you want to read out your question? Alright, read it for you. Or any folks considering you're doing the migration without a redesign, more or less the same UX, but the back end is no denying. I've never seen that happen. People often say that's what they want. But it always ends up being more complicated than that. It doesn't look like David is still in the room. OK. And Joe posted a link to the upgrade status module, which I think I also paste it into the shared doc. (INAUDIBLE), how long will this chair (INAUDIBLE) available?

It's supposed to live as long as the internet leaps. But we might, we might create a backup, just in case. OK, this is quiet, I'll go again. Another thing that I like to use migrate for is, you know how, for some entity items, you know, for menu config, you can export that as configuration. But for content based menu entries, you can't export those. So, something I use migrate for during my, you know, my script for site setup is having a migration that, you know, just sets up all of you, your menus in it from a source data file. That includes all the text links, you know, permissions and setups and everything. So, rather than use something like the default content module, we'll use the migrate API? I find migrates so much easier to use the default content. And then what kind of source do you use? And (INAUDIBLE) mostly, but it doesn't have to be anytime something gets more complicated. It just needs a more complicated source. But for almost everything that I do I just use a (INAUDIBLE).

Is it possible to share screens? Sure. Sure. OK, I didn't quite find the (INAUDIBLE) was a tweet, but I have another one that I can share quickly. Can you see the screen? So, basically, this is what I was talking about before, if you want to migrate anything, it's good to understand the relationship between the different entities. So, for example, let's say that you have a node and that node has three or four or five different fields attached to it, some of those fields are going to leave just right in the node. For example, if you have the title, the description, the body field, that's normally when you fetch the information from your source, that is going to be a touch right there directly. But in some cases, like with taxonomy terms, or images, or paragraph or related things, the node only store a reference to that or entity. So, for example, from your node, you have references to users, you have references to files, you have references to paragraph and so one, two taxonomy terms. So, understanding that relationship is going to help you like guide the planning process.

For the most part, as I said before, you're going to want separate migration for each of these different entities. One for node, one for users, one for (INAUDIBLE), one for phase and one for paragraph. And in some cases, if the, if the migrations are complicated, well, if none of them are so complicated, like if the if for example, you have multiple paragraphs, and that the structure among them is very different from one another, you might even go as far as like writing one migration pair paragraph type. Sometimes the same way that you will write one migration for each content type, you might do similarly for each paragraph. And basically this is like, sometimes you will have one migration for one entity. And sometimes you are going to have one migration for each bundle of the entity. So, for each paragraph type, for each content type. Sometimes for each vocabulary if that, if that into (INAUDIBLE), but basically, it's like a thing was Joe, who said that, if you can come up with a spreadsheet, with (INAUDIBLE) all your fields, and pay close attention to the ones that are referenced fields, you need to, you know, to write separate migrations for those, for the most part.

One caveat with a paragraph is that those are entity reference revisions. So, you actually need two IDs to make the connection. That's something that's the nature of how that works. But you know, that's the main idea. And from there, you know, this is another example, if you want to make it into media, you know, you have your note, you have the reference to the media, and then from media, you have the reference to the file. So, if you're able to build this mental model, you know, either head or in a piece of paper, in a spreadsheet or something like that, it is going to help you with, you know, planning and organising and knowing how many migrations, you have to write. I'm going to look for a tweet with this screenshot to share. But yeah, that's basically what I wanted to share. Thanks, Mauricio, that's really helpful. And I had a recent project where they had a bunch of, it was Drupal seven, Drupal eight, migration, they had a bunch of nodes, that all had the same structure in their body field.

So, they didn't have a separate image field attached to the node. But everybody field started off with an image and some sort of fixed text. And so, when we migrated to Drupal eight, I wanted to make it more structured, pull out those images, and attach the images as a separate field, and then be able to theme it more consistently. So, that was the reason I wrote two new process plugins, one of which pulls out the URL from the image field, so that you can use that to populate a separate field. And so, it pulls out the URL from the body field. So, you can use that to populate the image field. And then the second one that removes it from the body field, so that you don't have two copies of it now that it's in a separate image field you removed from the body field. So, those are both now in the migrate plus module, but currently only in the depth version, until the next release comes out. They don't get re embedded in the body field, right? They're just separate fields now. Yeah, now they're separate fields. And the theming of that content type, puts them in the right place.

Make some responses and all that good stuff. You can probably do something similar, where you find that URL in the body field and use it to generate a media entity. And then in the body field, you could replace the URL with like the Drupal media embed tag during the migration. Yeah, there are modules that help with that. I have trouble keeping them straight. There's one migrate media handler. And the other is media migrate. And actually, there was just some discussion in this in the last week that the maintainers of those modules are talking about joining forces. And then once they do that, I won't have to keep constraints. So, I have a question, is there any way to migrate web forms? Or is that something that don't even bother with that just start from scratch if you're, if you're moving Drupal seven website up to nine? As far as I know, there's no way to migrate the web forms themselves. You can migrate the submissions. I've done that on at least one project. The other thing, there's no automatic migration for his views.

And that's just because there are so many modules that provide additional functionality to views, that it's a really hard problem to do it in any sort of reliable and general way. That's another thing we usually make a spreadsheet of, like, all of the views, and all of the web forms are those things that like, you have to rebuild in the updated site? And we use that as a way to figure out like, are we actually using this view? Do we need to recreate this one? Or is this a good time for us to delete it and pretend like it never existed? Yeah, that's a good point. Any of the easy views are all easy to recreate, all of the more complicated ones, you know, you look at them a year or two after you wrote them, and then you just like, Oh, no, I have to, I have to make that better. I'm not just going to migrate that; I need to rebuild it. I (INAUDIBLE) the link to the photos. It's actually a tweet with the photos add to the document. I guess we are on time, but it is also lunch. So, we might as well to stay a little bit longer if we want to continue the conversation, otherwise, thank you very much for joining and sharing your experience.