Feeds UI + Migrate Engine = Dream Migrations and Imports

hello hi everyone and welcome to our

first afternoon session press the red

button so hopefully according the topic

of my presentation is how to do

migrations with ease we just had a

wonderful wonderful presentation about

how to write migrations and as great as

it is it seems a little bit very high

entry barriers so before I can learn

this migration it's gonna take a long

time and maybe by that time people will

be out of business altogether so today

I'm gonna talk about the way how to

makes things easier and one of the big

goals is to increase Drupal adoption and

hopefully the conversation would lead to

having the module that allows content

managers work with migrations in a more

efficient way and wider Drupal adoption

I with developer and open source

evangelist today I'm gonna talk about

what content tools are available for

Drupal 8 I'm gonna concentrate on

comparing what a seaman was different in

the feeds module which is one of the top

which is now in core which is a huge

change and gives a lot of new directions

for development and then I would like to

present a new module that were working

on and invite everyone to use it try to

contribute it evaluated give us the

feedback and the hope is that at bat

camp who will be able to present

actually working module and first

migrations done with feed migrate we're

using dev tools things will break be

aware so before I dive into migrations I

know that everyone had been working with

the websites but I like to put text

things in context

websites don't exist forever and they

change as change organizations that they

support and that is again this is our

angle our end users we don't want to

forget about it when we're very busy

coding this excellent wonderful

migration except that nobody can use

except for us and we've done that what

we're moving the round couple of

definitions we're moving structured data

from files from a other websites and we

have different goals sometimes when we

bring in something from the old

scientists at one time paying some

things to want to pull in on the regular

basis like news changes profile stuff

like this and you want to be able to

have easy ways to monitor what's going

on and of course now so what tools do we

have to reach our goals I'm gonna

concentrate on three main things that we

have right now and what we would like to

have all these three things I'm going to

be demoing and when I am jamming we keep

in mind different scenarios sometimes

we'll start with a data source we have a

website it has structure data we know

what people are going to be bringing in

so this is one scenario to have data

reformat and then put it into

destination container that would build

based on what we already have this works

great like when you have an old site or

removing it today new site some types of

work in Reverse we have existing website

where we want to aggregate different

news or different information or you

know new products or whatever we do this

is when we have preset destination

container and we want in we can tell

people format your data this is how you

need to format and then we will bring it

in with one-click this works well for

imports and ongoing everyday

so if we're ready we're gonna start

looking at all this tools I'm gonna do

this demos reasonably quickly without

much details because they have been

described in tonight in many places many

times but if that's the context for the

conversation so this is our this is

scenario when we have existing website

this is our website we have director of

different doctors in different units

with different roles we're hosting our

site on Pantheon Pantheon thank you very

much for ongoing support and like it's

been great always and we know our login

credentials so I'm gonna simply run

where did I put it here I guess so I'm

gonna read the manual because things are

gonna be exactly the way it says in the

manual I'm gonna log into my existing

database shit that I'm I have correct

password and forge 23 to 76

and see all the rights words and

hopefully things will work as expected

we'll see in a minute

sorry it is live damn but this you know

do we have to prove the point we have to

eat our own duck foot if we can't run

the damn leg what is this worth so while

Drupal is thinking we have time to you

know talk walk have coffee have

discussions and talk about what is great

about this upgrade because in the

pattern we're doing six to seven we had

to set up all the content types

we had to create data expert data import

this upgrade button saves me three

operations and he would convert it to

dollars it's going to be quite a large

amount as we said reading manual is very

important and of course things don't

connect so we're gonna use gonna give it

another try

Pantheon Pantheon database password

maybe I didn't put the right password

let's double shift to three two seven

six but if it doesn't work

two three two seven six

then we will use our imagination and

blame it on the network and to move on

to the site where we only did run this

migration

so while migration is happening what is

happening in the back is the drupal is

setting your migration and keeps

reporting whatever it is doing in the

Condor riding screen if it works and if

it doesn't work then it writes it in the

log and it is installing your modules

and then it starts configuring that

migration that we heard about in the

previous presentation it's setting all

the yellowed files it's setting all the

connections and then it starts up

greeting menu items image Styles

taxonomy is everything has been abraded

behind the scenes and when you're done

you have all your content moved very

nicely from this Drupal 7 directory into

into this Drupal 8 installation

everything has been preserved except for

some you know special things that we can

talk about at a different time epsilon

is our set yields are set the only

challenge is that if I made something

make changes in my original site I don't

have a way to bring them in again

that is a one-time thing okay so this is

not a really ideal tool for me when I'm

working through ongoing changes stuff so

I'm looking so this is one-time thing

and then you have to be very particular

about different setups so my next line

of defense is configuring input with

feeds which is similar to what we're

doing six to seven but how many people

are familiar with feeds okay so we don't

need to dive into the detail discussion

is no that is exactly what you don't

need to do so what you need to have you

need to have modules that you're going

to be using no because this is the power

of this migrated core that already have

in six and seven there was feeds module

and there was migrated module this were

two full stacks of everything to do my

to move things around and on average

people besides it has more site building

and content people they would use feeds

and people who use more developers would

do migrate that's how it work because

Interpol ate my grade is a poor it makes

the case of for using feeds as a you

more of as a UI and not building all

this huge engine and parsing and

processors that were sitting in the

feeds in Drupal 7 and that's what we're

gonna talk about so

for for the feeds part is there somebody

who does not know how to it's world so

the important thing about feeds I'm

gonna leave more time for discussion and

skip through things that everyone knows

the important part about feeds is your

first configure feed type you have UI to

put in all this information and then you

can do mapping and then you can do

tampering this is how you configure

migration so if we're now talking

migrate language we're setting migration

where the data source is now you cannot

do it with mikrotik normally don't don't

do it with migrate module okay so in

migrate module you know what's your data

source you build migration and then you

have the destination in feed opposed the

other direction we built the container

with predefined data structure and then

we say now I can start importing all

kind of reuse like for example this news

I am going to import from this file if I

wanted to say let's say this our news

from and why you and why you write then

I'm going to be this is going to be

something that

but time to news aggregation from

different university I will say okay

this our news from Stanford right I can

upload another file or provide different

HTTP and they all come in in the same

place I don't need a developer things

happen real quickly and it is a very

efficient module and so our empty users

and again I'm representing end users and

then the next you know there's an end

user there's content managers and I'm a

site builder so in this food chain

developers are the next ones was like I

need from this is what I need from you

developers so we can serve our end users

and Drupal will be as popular as

WordPress this is you know there's this

chase that keeps going on so the migrate

module also gives you some UI options

there's more and more options coming in

but then we're not really designed to

serve this population of site builders

and content editors so this is an

example of how you can run Josh define

death in bring in your configuration

then you can do execute however it

doesn't work for that users and this is

why we came with this idea that feeds

and migrate can work together and we're

going to take the best of both worlds

and hopefully it will work for everyone

and everyone will live happily ever

after

so this is like nice stories almost

Alice in Wonderland and this is what the

things that we're trying to bring in we

already talked about how they they go

the opposite direction and thank you

very much to people on slack if you can

hear me from the

and for everyone who contributed their

their time to early version so I did

this presentation like four times now

and every presentation brings in a

little bit more a little bit more a

little bit more so at Drupal pawn a year

ago Mike Decker fold it first Emma so I

was able to show it to Lucas and to you

live on Kapil they got together said

this is a good development path change

Dixon I joined us and now we have this

huge channel which is happening right

now very conveniently scheduled where we

have weekly meeting approximable at the

same time where people can ask questions

about feeds talk to whatever is

happening and there's something

happening right now and I don't know if

mike is around but hopefully he is so at

bath camp with it powerpoints release

and then their people from marching from

tableau also joined the team and today

I'm gonna hopefully demo this new

project let's welcome this new baby

project is on drupal.org by the most

recent code is on github or go back and

forth there is slack channel their skin

we even have like this all the HR

managed project management tools here so

well this said I am going to

hopefully define new migration so first

I enabled module feed migrate and

Fitzhugh I fit my great UI so I'm going

into my normal migrations tab I defined

migration group for whatever and this is

how I am adding migration let's say I

already have news defined for let's look

at the people I can't really execute

migration from here we will have to make

a lot of changes to this UI because some

of these things belong actually to

migrate but they will not work with this

configuration because migrations are not

executed in this part but you can edit

this Mike migration and for those who

use migration on the daily basis this is

the part that is UI that generates your

llamo files but sir I can define here

basic settings and this is big new tab

that did not exist before and here I can

say I want to map

body and the source will be say in the

in my CF CSV or whatever I'm using

gladsome how it will be by a tab things

that we are need a lot of warp on would

be setting process plugins that are not

there yet and because they're not there

yet

I can do actually import today but I

hope that with all the enthusiasm and

contributions that I have seen and

hopefully maybe there will be a buffer

spring tomorrow maybe things will get

moving quicker and once I defined my

migration this entity doesn't really

have a full name yet I can come here and

I can say I would like to import I will

be imported directors and surgeons I can

edit it I can say you know what I want

to replace existing content instead of

editing these are things that are done

on different basis for each feed

importer because this is Visser dis

decisions that are made by content

people not by developers and then I can

say okay I've got the new source

hopefully I will find it where it's

supposed to be XML and I'm gonna save

and import

and mm that input field and I already

know that and one of the reasons when

I'm doing this presentation inviting

everyone is contribute because we need

more people to make this connection um

so this is kind of in the presentation

you can take home and see at the screens

that are supposed to say what it how it

actually should work there's couple of

items that are clear to define as they

either work in progress or things that

we do want on the UI but they're not

there and yes it is compatible with

current migration so you have migration

set up on on your system it will show up

to today in the UI and you can pick you

can create new importer from the old

migration the only problem is like

usually in the old migration you will

have source predefined and we don't have

right now mechanism they want to

override it they will want to replace it

do it what just not show migrations that

have hard code hard coded source these

are all things for to do to brainstorm

to be an architect and to do all these

great things so this is so this is one

of the things that are like our prime

target and then this is another wishlist

item

in the similar way like if you have data

source that you already know that

already shows your data structure and

you want to build up all the all the

mapping in the similar way to migration

it would be great if it could be done

eventually when and how we don't know

yet plugins tampers work in progress it

would be great to have one screen where

all you can go through all the plugins

and check them out instead of going down

one by one in a field and we hope that

eventually we might have it be the

processor is done in the way similar to

the way web form does it when you have

this beautiful flyout hx4 and just like

everything is on the same screen very

cool so this are things that are very

specific to feeds but are not I don't

know are they defined in migrate like

how often do you want to run migrations

do you usually define it well so this is

why I like with are things that you can

make different decisions on different

content units but for example we want to

pull data from Yahoo every day and we

want to pull from dipole news once a

week because we know they're not updated

we care about some other things and then

of course we want to be able to very

carefully roll back delete or stop

migrations

so this is what we're doing but if there

are developers who would like to

participate we'll cordially invite them

if there are companies that are

interested in being first users trying

saying all this will work better for us

this is where we can do things this is

what is this presentation about to bring

people together and to make things

happen and where I can spend more time

in the details but I would rather he

like in the question/answer mo manages

demo mode

questions anyone

so the selectors

you can select the options is that just

pulling in whatever is available for

that current site did you created a new

process plug-in or something would it

show up as an option yes it would

this is like this is very similar to the

feeds when you are adding they feed let

me real quickly add the feed because

this is all about the process and this

whole thing is about the details so if

we are adding I have 15 sites open this

one so this is the feed right so if

we're adding feed structures feed struck

fit type we can we can add different

tempers is it are you asking more about

tempers or more about something so bone

mapping happens from data source to

content type to the field of the content

type one of the big challenges is how do

we import paragraph in how would you or

migrating field collections because

there is no field collection of great

automatic upgrade paths and this is a

huge problem so all the field

collections now has to be migrated one

way or another anyone using field

collections in Drupal 8

anyone using paragraphs in Drupal 8 so

more or less path is from field

collection to paragraphs paragraphs as I

learned have have slightly different

structure than field collections they

have more connections to parent parent

node and that makes my effort just plain

import a little bit more complicated

because you can Newport paragraph but

they also need to be correctly

identified in the parent node and it's

not happening right now so these are big

challenges that we're facing and we

don't have answers today so to answer

your question in defeats we have source

and then we have destination this is

very similar to to migrate but say we

want let's say we're getting time time

that note was created from Drupal 7 in

not so Drupal 8 is is so time Drupal 7

is not so there it has to be a converter

that will put letter T in the middle so

Drupal 8 can correctly pick up the date

this is where either you have to write

my great plugin which we have to do a

year ago and it took a lot of time and a

lot of money from our user which we

don't want to do or we could potentially

have here plug in then would say convert

something to something and here it is so

this is country to ISO code which is not

what we need to do it will come convert

strings to you next time step right so

here you can write a limited number of

plugins if you have very custom JJJ 30 I

store all this structure in this

elemental weather except similar Jason

and this is my very unique processor

this is where you will plot it

through the you are so that gives great

opportunities both for developers to do

what they need to do in code within my

grade and for end-user to say this is

what I want this is con process does it

answer the question

so you can see that here we have very

limited amount of processors available

for feeds now because coding every nest

processor takes time and there's not

that much time in every day and there's

not that many people who know how to do

that in migrate will already have it so

if we can have this structure so right

now we're putting it in one of

migrations group migrate so here we are

putting it into migration speed

migrations there would be nice if we had

direct like drop-down to come in here

but here when we're editing this so this

is slightly different interface we have

option to add process plugins on every

it gives more room the hope eventually

to compress it into a more like what

form type thing where you have a jobs

fly outages okay we have configuring

everything on the same basis there's

less clicks but for now we need to have

at least this thing working and so this

is where your plugins will come in and

we have fearless leaders that will tell

us how to do batch and that's not me

look as I am looking at you

so we hope that you will we can have

like if there is enough developers who

are interested in doing it we could

probably have another workshop organized

we did it last time remotely about a

Europe like it was after Nashville it

was in June so we did the workshop where

Lucas showed whoever was available at

that time how to write these plugins and

the bed wasn't what is it was something

else

but that gave it start to developers who

needed some training or to save the time

and we'll be happy to sponsor this

workshop there's enough interest

because fear we would really like to see

this happen

this means they were over overly

disciplined

we're not you ask people when I do I

people there's a reason why I've been

doing drugs for so long because they

like to cover so we really want to see

the community pull together and build a

really good user experience for

importing data that's my plea

we make the changes given contract and

if we can't convince our alter egos in

core if we need to to make those changes

to support things like this yes thank

you

thank you very much because we already

have this been happening last year I

remember then so maintain our feeds

yerevan cotton and James Dixon and

marching from top law this is a running

conversation on the feed channel this is

where conversations are happening

everyone is welcome to join and login

it's on the main Drupal and it's on the

feeds channel I remember that last year

there was something not working

correctly we needed some change in the

migrate code so Lucas will know better

than I what has been changed but it

happened and we didn't have to wait for

three years

it just happens like this and I think

this is one of the great things that I

want to think what does that all the

migrate people who are supporting and

making this happen

because from the UI perspective we have

very very very clear understanding of

what we want to see and we can keep

putting up this screen this once anytime

you're not sure how to code and where to

place an email me 24 about seven I'll be

happy to tell you that this like this a

search does not exist this is just a

Photoshop then we said it would be

really great if I can do search from my

this content type they never fit on the

same page and in one pontoon type I have

have there and every time I need to do a

temper and I miss it by one I have to

redo it again the search would be great

but it's not there yet so we're looking

at people who write to code who I'd rush

for white command wire to join us and

then everyone will be happy

yes

it sounds like there's a few different

pieces that are kind of like you can use

some of them you mentioned like you

could write my greatness and then this

could be a way of just schedule them or

paint the way that the source works on

fly

it sounds like some of this is evolving

but it's never like what is the primary

use case for like are you intending

excuse all of them present incentive to

just like pick pieces that maybe you

would use the UI to build the migration

inscription shouldn't used to do like

run it all the time like is there a

singular like main juice came for this

or is

so the original driver was to not

reproduce all the code for XML and Jason

that we already have it for CSV is not

on core so feeds that are working with

CSV is very important piece for people

who are working with important Excel

files this is a huge audience

but right now feeds do not work with XML

or JSON and so people who are doing that

rotation of content they have to you

know work around it there are ways to do

it

and that's okay so we're looking at

feeds more nodes in Haven even more so

there is feeds extensible parser which

at this point is basically reproducing

things that are already sitting in court

so that was original intent to reuse

what's in there the use cases for

migrate versus feeds can but there's

like huge variety so it's not like it's

all designed for our ongoing efforts or

it's only designed for one-time thing I

think we advantage of having this UI it

has all the flexibility to build

slightly different your eyes without

investing tonne of development time

every time there is an interesting

module that was recently developed by

Adam group called import source import

we didn't I didn't test it thoroughly

yet but that is another piece that also

allows you to build migrations via yo I

somewhat but that was kind of more

limited to CSV and is not covering all

the cases did it answer your question do

you want to see some examples

I think it make sense yeah it sounds

like the

and the UI even

yeah we would like to I strongly believe

that if we can get this first

you know after release then everyone

will have an opportunity to take it and

actually use it then lots of companies

large companies will say oh okay great

that's exactly what we needed we need to

tweak a little bit here tweaked a little

bit there so this is like pioneers and

avant-garde developers please join with

you know go out there Wild West

so if a developer wants to jump in and

work on an issue do you have some issues

ready to be worked on yes this is the

best thing I could have hurt dear

developers let me introduce you to our

very project manager management type

words Board of issues that he can work

on however if you want to go to more

familiar more familiar route you can go

here and we have reasonably long list of

issues the goal is to have them more or

less well-defined because if you it's

very difficult to go to chew them all

all together so if I will find where we

have all issues then I'm happy to talk

about them one by one if you have

questions

[Music]

so like this is a very simple request

that will eliminate need to do two

clicks right now I need to go to

migrations that I need to add migration

group and then list migrations if we can

have this feed important migration group

preset in the module and have URL this

URL pointing there that would be great

it's a reasonably small task for a

developer you know and it can be done in

limited amount of time without full

understanding of every migrate API some

other issues require much more detailed

conversation if you have questions again

feel free to email me I don't have the

answers but I connect you to can connect

you to the person in this room who knows

the answer a person on squad channel who

knows the answer work make sure that you

get your answers

and the goal is to release kids migrate

and Vicki there will be both on

DrupalCon so this is the schedule

so this is kind of a schedule for next

couple of months anyone going to

DrupalCon

this year so James Dixon will be there

and they're scheduled off and I would

like to invite everyone to come visit us

at Stanford

we're gonna have Stanford Drupal camp

this year and me it would be great to

see you later and then they're going to

be some stuff happening in Seattle

but yeah this are the issues that have

been identified and described in his

holders

I can talk about different use cases or

whatever it might be of interest to

people in this room is generic

well it should work for the idea is it

should work for any eight and higher and

we will make sure that in nine it also

works because some people are at this

point are not migrating to eight waiting

for a night so they will go seven

tonight

so yeah the plan has to work with all

Drupal 8 and higher versions that's the

goal it's a different presentation I

don't know it partially depends whether

this module will be ready or not because

for us migration right now we can't run

migrations to Drupal 8 because we don't

have enough tools to move content

efficiently and so for me

this model was part of for a Drupal 8

adoption story

this was my blocker to move like half of

myself the sites that we maintain like I

can't move like it we're still building

new sites in seven because there are

like we can't import efficiently

paragraphs into Drupal 8 and so we're

saying ok we're going to build in seven

and then wait until night unless we have

this module ready before we need to

launch the site and then we can do it so

it's like challenging question

okay any more questions

[Applause]