Federated Search with Drupal, Solr, and React

[Matt] Hey everyone,

we're here to talk about Federated Search

with Drupal, Solr, and React.

And just a little bit about that.

This is sort of our solution
for the void that was left

in the Google Search
appliance and things like that

they stopped supporting that,

so a lot of our clients had to figure out

what they were gonna do
with their search solutions,

and this is what we came up with.

So just a bit on what
we're gonna be doing today,

obviously introductions,

we're gonna do a project overview,

why did we build this and
what exactly did we build.

I'm gonna show you it.

We're gonna show you a
little bit on how it works,

we're not gonna go too in depth,

we're just gonna explain the components.

And we're gonna show you how you

can get started using it on your sites,

and then answer any questions
that you guys have at the end,

hopefully answer those questions.

So Dan and I both work for Palantir.net,

Palantir.net is a digital consultancy

who provides web strategy, design,

develop and development
services to clients

and higher ed healthcare,
government and beyond.

So that's our team up there.

And my name is Matt Carmichael,

I am an engineer at Palantir.net,

I've been working with Drupal
for about three years now.

I mostly do backend work,

I do a little bit of front end work.

This project actually got me
really interested in React,

that's one of the components that it uses,

so I've been trying to learn a lot

about that lately as well.

- [Dan] And I am Dan Montgomery,

I am a senior engineer and
technical architect at Palantir.

When I get a chance I like to
work on React a lot as well,

so excited to be here.

- [Matt] So project overview,

as we just alluded to,

this project uses React

to actually display our search results,

it uses Solr as the index,

and then obviously Drupal
to connect the two,

and we'll go into that a little

bit more throughout this presentation.

But we have been working with the

University of Michigan for a while now,

specifically on their healthcare sites,

so they have a handful of Drupal 7 sites

and a couple of Drupal 8 sites,

I think it's something like six

and two or something like that.

But essentially they
have a few Drupal sites

that they were using
Drupal Search Appliance

to get search results from all

of the sites in one result set.

And then when Google
end-of-lifed that project

they were left with a
void of what do we do.

We want that same functionality

but we are not sure where
to get it or how to get it,

so they came to us, us being Palantir.

Our team did a spike

and realized that it's not that difficult

to actually index multiple sites

into the single Solr server.

The issue that they had was

how do you take these sites

that have different content models,

that have different content types,

the taxonomies don't align.

How do they take all
that and put that into

a unified index.

And the other issue was

how do you display those
results consistently

In a way that matches the theme obviously,

but how do you display those

consistently across all of the sites.

So like I said,

one of our team members took a few days

to run a spike on that
and figure things out,

and the Solr--

- [Man] What do you mean by a spike?

- [Matt] What was that?

- [Man] What do you mean by a spike?

- [Matt] Sorry,

he just spun up a couple of Drupal sites

and just dug in there and
just messed around with things

and figured out what
solution would be viable.

- [Man] Like a research spike?

- [Matt] Right,

so just figuring out how we could solve

this problem for the client.

And this presentation
is what we came up with.

So essentially what they want is Google.

It's tough to build
Google believe it or not,

so we came up with our
best solution for it,

and one of the tough things

was how Google crawls the websites,

we didn't have the time
to write the crawler,

So we leveraged Drupal in a couple of ways

to make that a little bit easier to set up

config that will,

as opposed to crawlers crawling a site,

you can map how things will
map into the index yourself,

and then after that it just
does the work in indexing.

So again they had the Google

Search Appliance that was end-of-life,

and then there was Google Site Search

that they also discontinued
when they stopped supporting.

- [Dan] I think one of
them is you can purchase

a physical hardware
and host it internally,

and the other one is
like a similar product,

but I think it's hosted for you,

that's about the main difference.

- [Matt] Yeah, so that's
what I understand.

So after breaking down their needs

and thinking about what
solution would work,

these are the parameters

for the unified search
solution that we came up with.

So one, we needed a simple way to start

retrieving and parsing the data.

Obviously all search
applications need that.

They needed it to be cross-platform

because they have Drupal

that they wanted to be able
to search all from one index.

And then they needed it to be speedy,

usable and responsive,

that's where the React side comes in,

we'll show you in the demo in a bit,

but it's super fast, super usable.

And then it obviously
needed to be flexible

and extensible and reusable.

We didn't want it to be something

that you just drop in
and it is what it is,

display wise and things
like that with the facets,

so we made sure that all of the sites

can edit it how they
needed to lift and work.

And then obviously we wanted it to be

a drop-in replacement
for the Google projects,

and it's pretty easy to get started on,

and then you can customize it as needed,

so we'll show you that as well.

- [Man] How did you get to the list,

the differences between the things,

was it something that you did on your own,

or did you have interviews with--

- [Matt] What do you mean the differences?

- [Man] So you said there
was six or seven sites

you were trying to implement this on,

with different stuff,

did you have to figure that out yourself

or did you go through some sort of process

where you were interviewing
or gathering data

from people at Michigan?

- [Dan] Can you repeat the
question for the recording?

- [Matt] So the question was

how did we figure out the differences

between the different sites

And how we would have to deal with those

when putting them into the single index.

The way that we did that is by

we did a lot of research on
just general search applications

and search boxes throughout the web,

so what works best for people,

what results display best
and the best user experience.

And then we actually built
this application in a way

where you have the default
fields that you will see,

and then you're able on
the Drupal side of things

map each content type.

So for example, there's
a field Federated Date,

so that will add a facet for date,

and then you can decide are we
gonna use the published date

are we gonna use the data
created, things like that,

last updated.

And we're gonna show that in a bit here.

- [Man] Alright, I'm
getting ahead of myself.

- [Dan] Yeah I think we're,

so there was a couple of different stages

of user testing and collaboration,

so that part there was some
user testing of the application

to see what people wanted to use the most,

or how they wanted to use it.

There was also I think
a lot of conversations

with the University of Michigan,

so Avi might have more information.

- [Avi] We had a strategist come in

and work with the client,

this strategist went through all the sites

and took an inventory of
all the fields in the sites

and then worked with the
client to reconcile all of that

and figure out what the best way

to put it all together would be.

- [Man] Cool.

- [Avi] So it was both
an internal discovery

and an external discovery process.

- [Dan] Yeah so for the recording,

what Avi was mentioning was
there was a strategy phase

at the beginning of the project

where they canvassed all the sites

And did architectural work

to try to figure out how to map

those things for the customer.

Cool.

- [Man] Cool.

- [Dan] That's a very good question.

We're trying to deal
with some screen stuff

for a second here,

but I think could be bring it back?

I think you can just,

yeah just check it, it should work.

Cool.

- [Matt] So now we're gonna take a minute

to give you a quick live demo.

Sometimes it's hard to follow
words about these things

so we're just gonna dive in
and show you how it works.

So this, do you wanna explain this?

- [Dan] Sure, so this
is a Drupal 8 website,

so we have a demo repo where

we put a bunch of different sites,

this application works
on Drupal 7 and Drupal 8,

so this demo repo has some internal tools

that Palantir reduces.

Is this kind of small.

It's basically a search box.

So the demo repo is a public
repo that anybody can download,

you can spin this up,

you get a Drupal 7 site,
you get a Drupal 8 site,

you get default content,

and then basically for Drupal 8

you get this Umami theme,

which seems like everybody's using

and is super helpful for demos

because you don't have to
make something for yourself.

And the Drupal 7 site uses

I think develops and likes

to just create some random content.

It also includes a Solr server,

and so we have all that running,

and this is gonna show how this works.

So since it's the Umami site

you could do a search for pasta,

you could see very quickly,

I mean this is all local,

but very quickly the results show up.

And I'm gonna make this
a little bit bigger.

So you can see the search
results show up very quickly

with a lot of the common
features like sorting,

and the counts, the filters
have been updated on the left.

This is all happening without page reload

since its a React app.

And then you can start
to see the search results

and we've tagged the sites here

to just indicate that this is
coming from this demo site,

it's using a Drupal 8 site
that's using domain access,

so we wanted to make sure that it worked

with those models as well.

And then we are actually on the site here,

so this piece of content

is from the current site that we are on,

and this other piece of
content is from another site

that's also running on
the demo environment.

So you're seeing recipes, articles,

it's doing the highlighting,

a lot of the stuff you
expect out of search results,

and this is all doing
it without page reload.

So there's also facets,

so there's a couple of
default facets that come,

with the recent version you can disable

the default if you don't want to use them.

The defaults include things like,

that got formatted a little weird,

but there is minimal
styling out of the box,

sometimes it looks better than that,

and sometimes it looks a little weird.

So you're seeing the same
sites that I mentioned before,

and then if you select them

it immediately adds that as a filter

and the results get updated.

So if anybody is familiar,

I think we are all somewhat
familiar with the React stuff,

it happens immediately,

because it already has a lot of this data.

And so you get very quick responses.

You can deselect that and
go back to where you were.

You can filter by content type,

I'm not sure what's going
on with these check boxes.

So you can filter by articles or recipes

that updates the other facets immediately

to only show things that are relevant.

Dates, they have filters.

And then some of the ones
that are not out of the box

but are configurable are things like,

in this case ingredients
doesn't apply to all websites,

but it does on this one.

We've added one for pasta,

this is the only one that's
been tagged with this,

so we can talk about
that a little bit more.

But the way that these extra filters work

is a little bit more hands-on,

but it's built in a flexible way

to support any kind of site.

So for this Umami site,

we were able to configure that

and add a facet for ingredients,

and then figure out how that works.

But on a site like University
of Michigan Health,

it doesn't have those same filters.

So it has a lot of, sorry--

- [Matt] We had a question.

- [Man] So if we take a look at this page,

is the filtered part of the React app,

what's React on there,

and what's just performing
in the background?

- [Matt] So everything you see is React,

essentially the way it's displayed--

- [Dan] Except for this.

- [Matt] Except for, yeah.

So the React is embedded
in the Drupal site,

all of the search elements are, yeah.

- [Man] So basically the
content frame is all React

and the rest of the
container it's in our Drupal?

- [Matt] Right, so we're
gonna show you a bit later,

all of the markup in this technical file

is just a div with an ID of route,

and then the React app looks for that div

and just dumps all of
its display in there.

- [Man] Everything below the breadcrumbs.

- [Matt] Right.

- [Man] Is the experience
for the user any different

if you're logged in

or either you're tagging
with protected content,

or anything or is it
just the same experience

whether you are logged in or not?

- [Dan] In terms of the
search results page?

- [Man] The assets, the
display, the saves favorite,

is anything different in
terms of functionality?

- [Dan] Right now there is no
authenticated user addition,

so this is the same experience.

I'll actually come back to
that I think in a minute

about what this means,

well I can just jump there.

Like you could put this on any website,

this part of it is not Drupal specific,

there are some things to make the Drupal

implementation easier to set up,

but you could put this
React app on any website

and it would still pull these
results in and display them.

So if you have WordPress
sites in your network

and you wanted to show similar
results, you could do that.

This part is not Drupal specific.

- [Man] The second part is
with the different sites

do you have the capability
to go across sites,

or is it all whatever
site you're on that's

what it's searching and that's--

- [Dan] No, so in this case

this is an example where this is,

I'll just click on them,

this is one website and
this is a different website.

So I know that they look pretty similar,

but let me do a better search

and do something that is
clearly not that site.

So this is the Drupal 7 site.

It just uses that develop content,

so this is an article
on the Drupal 7 site.

So this was, if you can see
it's very small I think,

but the domain is D8-1,

which is that domain access website.

This one is just D8
which is a single site,

and then this website is
the Drupal 7 site, D7.

So all of these results are
showing in the search results,

and that was the main use case.

There are features that if you want to

either by default or just period

only search one site for any given site,

those are options as well.

But the default,

I should have repeated the question,

the question was basically

are you limited to searching
results within one site.

So hopefully that answers that.

Yeah?

- [Matt] The results are just interacting

with the Solr server,

so it is independent of
what's going on in Drupal

which is what Dan mentioned,

how after the contents index

it's basically just Solr
and React interacting

and they are just displaying
on your Drupal site

using that template

that I just mentioned a little while ago.

- [Dan] Yeah, let's jump
into some of the specifics,

I think these are really good questions,

so if you have them just let us know

or we'll come back to that stuff too.

Okay, I don't know how to do that.

My skills with Google
Presentations are not always great.

So we've talked about already
the three components are

Solr, Drupal and React.

Solr is used as the search storage,

Drupal is used in this case to
create and send the content,

both Drupal 7 and Drupal

and React is used for the presentation.

There's a little bit more,

you could add Drupal again here,

so other versions of this talk

have talked about it as
the cyclical process.

If you want to know more
about that talk to Avi.

Where Drupal comes in again

where there is another Drupal module

that is then displaying
the search results.

So it's involved in the beginning

where its indexing content to Solr,

but then it's also using the
presentation part as well.

So we just have them listed here once,

but it's used in different ways.

So this is,

so Solr if you have used it before,

it's pretty flexible,

but there is a value in
having consistent ways

of storing data.

So you can set it up
where either it will just,

you specify exactly what this schema is

or the fields that your
gonna index in Solr,

and then you have to
create content for that.

You might be familiar a little bit,

if you have worked on Aquis search before,

when you've probably
never configured Solr,

or you may not have
configured Solr directly.

So Solr also supports a method

where if you send data

and you are telling it
fields it's never heard of,

it can just create
those fields on the fly.

In either of those situations

you probably wanna have some
sort of uniform structure,

and that uniform structure

is one of the bigger challenges

that we were trying to solve for.

And Matt has talked about
that a little before.

So what we're looking at here,

is that the index,

this is what the index looks
like at the end of the day

We came up with that uniform structure

where we said everything
needs certain fields

and then we can use those
fields in some flexible ways.

So you did see the facets,

and those were ways that

those were used a little more flexibly,

but everything needs
something like a title,

a date, a type,

so that was a content type.

A lot of content is gonna use images,

they don't have to use images,

but that is something that

a lot of the fields are gonna use.

Rendered items,

like what are we searching
through primarily,

so are we gonna create a
version of that content

that is the rendered
version of that content,

and then we need a URL,

because this could live on any website,

it's not just gonna live
on the same Drupal site.

And then site name,

in case you want to filter by site names,

and then the site.

- [Man] This is your entire (mumbles).

- [Matt] This is.

- [Dan] I guess I skipped
over terms, but yeah.

- [Matt] How it works is,

we'll explain how Drupal
is gonna feed into it.

Pretty much all of your content types

will have this data more or less in them,

and then we'll map to these exact fields.

- [Dan] Yeah,

and I think one of the
key points to look at here

is that none of this is Drupal specific,

so you can use different Drupal websites,

you can use those different
versions of Drupal websites

That have different fields,

but you want to create
this consistent format.

- [Man] So you guys don't,

there is not a description or body?

- [Dan] Yeah, so the equivalent
here is this rendered item,

and then in the search results before

it was using the highlight pattern,

where it was just
finding text that matched

and then bolding those words,

which I think were delivered
by Solr as part of.

Cool.

The question, sorry, I'm
gonna keep forgetting this.

The question was that there
is no body field listed here,

and how was that.

Cool.

So if you have used Solr
before, or if you haven't,

Solr provides this admin interface

which is very, very convenient

if you're doing local development,

and this is what it looks like.

This is what this slide
particularly is demonstrating,

is the way that Solr formats its data.

It might be a little bit hard to see,

but essentially Solr out-of-the-box

supports exporting data as JSON,

and that's a really convenient way

of importing data into
Reactive apps or other apps.

So Solr was a good solution,

because we were familiar with
it from the indexing part,

and also because it's very
good at sending the data back,

so getting data in and
sending the data out

is a great way to begin.

And you can see that those same fields

are the ones that are being listed here,

the structure is very simple,

it's basically a couple
of high-level fields

of how many results there were,

and then it just lists the documents,

Solr calls individual
pieces of content documents,

and then all the fields with their values.

Alright, so I think,

Matt?

- [Matt] So yeah that's an explanation

of the Solr side of things,

and then next we're gonna
look at React a little bit.

So actually we found this repo,

Solr faceted search React,

and we found that it actually did

a lot of the sort of work
that we wanted to do,

so this repo takes the Solr results

and displays them in a very rigid way,

there is one way that it
gets displayed and it works,

but it's not ideal I guess.

So we actually forked off of this repo

and then made some modifications,

made it a little bit more expandable,

made it theme-able,

and made it our own.

And like Dan said,

React loves to consume JSON,

So it works perfectly with Solr,

and this repo facilitated
a lot of that work for us.

So here's a look at that
repo that I just showed you.

You can see it works,

it has facets, it has your results.

Not formatted great but
it gets the job done.

So we saw this and thought,

this is a good base, let's build on it,

let's add some fields like images

to narrow things down a bit to our liking

to work with those fields
that Dan just showed you,

and add a little bit more complexity

to the options when dealing with filtering

and facets and things like that.

And it also is MIT licensed for reuse,

so that was good.

And just another note,

our project's no longer dependent on this,

so you don't really have to keep up

with what they are doing on
this project specifically,

we just used what they had

and we've been maintaining it,

so just a little note,

you don't really have
to keep track of that.

And then this is what we spun out of it,

you saw the live demo,

just going back and forth,

you can see they are pretty similar,

you have the facets on the left side

and the results on the right.

We just cleaned it up based on some

of that research that we mentioned earlier

on what people actually want
from a good search application.

So yeah, that is a little bit on

how we got to the React side of things.

And then again like I said,

this was designed to be skinned,

so you can make it look
however you need it to look,

it will be seamless with your site,

it won't look like you just dropped in

a big clunky search application

with table views and things like that.

And the markup that we use
is all semantic and easy,

so you can actually skin
this with just one CSS file,

just a single CSS file.

And as you can see the markup
should look pretty familiar,

and your team shouldn't really

have any issues looking through that.

So yeah, that's a bit on the React side.

Again we have the Solr side,
that's getting indexed,

you have all of your
results stored in Solr,

React is actually the part

that is taking those
results and displaying them,

kind of explaining again how Drupal

isn't really involved in that process.

It's just a facilitator.

- [Dan] Cool so,

so once we have,

so how Drupal is involved.

So Drupal we mentioned was involved

in indexing the results consistently

In this sense this is what
the document looks like,

it's a little bit larger now.

But we have covered this already,

but basically you don't have

to use you build index content,

if you're using WordPress
or some other system,

you can do that as well.

We haven't used a non-Drupal website yet,

but you could do that
and that's supported.

So if you want to fork this
for a non-Drupal website,

or add in support then
that would work as well.

As long as all the websites
are sending the same sorts

of requests to index documents to Solr,

then you should be good to go.

So this is the other part of Drupal,

so essentially when you're loading Drupal,

so to get this on your Drupal website

there is a module that
will help you configure it.

So it basically lets you say

this is the route that

I want this search page to show up on,

this is some of the configuration
options that I want,

and then it will create that page

and put that search block on there.

And there are some other ways

that you can go about configuring

the display of it with the styling

that we're not gonna cover now,

but those are all planned ways
that you can use this module

That you can add flexibility,

it's not at a predefined space,

you don't have to just copy the code

and load the library,

all of that is bundled
up into a Drupal module

to make that as a drop-in solution.

And then that works with the Drupal 7

and a Drupal 8 versions.

So you can see in this case

the configuration is being
passed into the React app

from that Drupal system

so it has the configuration
form and passed it in,

and then it's just
rendered as Matt mentioned,

it's rendering this
placeholder essentially,

that the JavaScript knows to
replace with the React app.

Cool.

So yeah, Drupal is not required,

but it's making it easier in this case,

since it's the one that's been supported.

Yep?

- [Man] I assume that only one site needs

this report search module in it

and the other ones that are feeding data,

well I guess maybe they do too?

- [Dan] Yeah, so the question was

which Drupal sites need what I think.

- [Man] So the React app
only needs to be in one,

even though it can serve
results from other Drupal sites.

The Drupal 7 site and all that,

it's not gonna run the React.

- [Dan] So in the example
for University of Michigan,

every website has a copy of those modules

because they are all
showing on search pages.

So yeah, those will all show them.

And then there's a set of modules,

there's a set of modules
that do the indexing,

sending the documents to Solr,

and then there's the modules
that display the results.

So depending on whether or not
you want to index from sites,

and or display results you
can do them independently.

- [Man] Excellent.

- [Dan] Cool.

- [Matt] So now we're gonna get
into a little bit of config,

I know there's been a
lot of questions on that.

How you actually set this up.

And we'll go through some examples,

and hopefully have a better
understanding of that.

So this depends on the
standard search API config,

that does a lot on the
Drupal side of things,

that sets up things for you to,

as far as setting up the Solr server

and indexes and things like that.

And then you're gonna
need a local Solr server,

only for development purposes,
but you definitely need that.

The alternative for
production sites would be

having independent Solr hosts,

so something like SearchDex
I think we use a lot,

or just hosting that elsewhere,

but for testing it's always nice

to have your local server setup.

- [Dan] Maybe as an aside,

so we are currently working

with better support for Aquis,

so the way that Aquis has Solr set up,

it has limited access to
only the Drupal application,

and as the React app needs

to access the Solr server quickly,

that is an issue.

So on the particular project

we used SearchDex as
an external Solr server

and SearchDex provides,

I'm not trying to give an
advertisement for SearchDex,

but it did work out well.

They provide access restriction,

I'm not sure the best way to say it,

but essentially you can
set up security measures

so that not anybody can just send content

to the search server,

and not anybody can just pull from it,

you can set up the permissions there,

and which sites are routes,

and which types of
users can access things.

So again, these are just the fields,

so you have date, image,
terms, title, type,

and then the HTML body,
the site and the site URL.

And this is just recapping that structure,

and then in the next couple of slides

we'll show how you actually map

your different content
types to these fields.

You always have this standard schema.

- [Dan] Yeah, if people are not familiar

this is the standard
Drupal 8 configuration

for how you set up Solr indexes,

so you turn it on and
you don't have anything

and you have to add any
fields that you want to add.

So you can specify labels,
you can specify machine names,

and then that helps
you build that process.

So this is what it looks like completed,

and Matt's gonna go over some

of these are very straightforward,

and some of them we're
gonna talk in more detail,

because of things like the
custom facets and other fields.

- [Matt] So let's just
dive into a quick example.

So here you have the
federated type index fields,

so this is gonna deal with content types,

so in the search results you can remember

within each result you saw a
content type above the title,

and this is the interface
where you specify

what you want that title to be.

So sometimes like we said

when you're coming from multiple sites

you have a bunch of
different content types

that might be similar,

but they have different
names on the different sites.

So for example here you can see,

you can see that we have the article

and we decided we wanted the display name

in your search for
assets to be blog article

and then if you jump down to say podcast

you can see that we
also chose blog article

for that content type,

and the reasoning there would be

you don't want facets for article, podcast

video all separate that your
user would have to click

when they're gonna get
similar content out of them.

So when you use the same
title for multiple types

that allows you to have one
checkbox that you would click,

and then you get a listing
of all of those articles,

so that was the thought process there

as to why you can set these,

as opposed to just coming in by default

with whatever content types

that you are using from the sites.

Does that make sense?

- [Man] Yep.

- [Matt] Cool.

And then just to jump back really quick,

these are all basic static values,

like you are setting them
and that's what they are.

Alternatively you can use tokens,

so similar to (mumbles)
or something like that,

you can see here for the date field,

federated date,

we have a majority of the nodes
just using the date created

because that's what makes
us most of the time.

But on articles maybe the date created

isn't as valuable as the
date that it was published

obviously to your site users

that's a date that's important to them.

So this just shows that you

can differentiate different
date fields as needed

and how to show that the tokening

system will also work here.

And you can see here at the bottom

we have the token browser
for you to go through

if you need any help with that.

And then the last example
I'll show you is the images.

And you can see this is also tokenized,

and it supports multiple media types

so you can decide which media type

that you want to display in

to I guess normalize your
image display results,

if that makes sense.

But here we are just using the search API

federated Solr image media type.

So you can see node, field, image,

and then the media type is search API

federated Solr image.

- [Man] Is that a media
type or a view mode.

- [Matt] Yeah the view
mode, sorry about that.

But that comes with the module

so that will allow you to
get some uniform results

with regards to the images.

And then one more example,

we have the site name field

and this allows you to enter

whatever site name you
want your users to see.

Sometimes as developers

we might not use the best site names

for our Drupal site names,

so this allows you to correct that

if you have not so
user-friendly site name.

And then as Dan mentioned,

this module works well with domain access,

Ken on our team,

Ken and Rickard does a lot
of work with domain access,

and as the project owner
of this specific project

he really pushed for that,

and the University of
Michigan used it as well,

so we had to make sure
that that was a feature.

But this was just
showing the configuration

if you're using domain
access on your sites.

- [Dan] If people aren't
familiar with domain access

it's a way of having multiple sites

live within one Drupal site,

so it looks to the outside world

as if it's maybe several different sites,

you're creating content
within that single site,

and then depending on
where they're coming from,

you'll see different versions
of the website essentially.

Yeah?

- [Man] If this was a
single site or a multi-site

this site name would just have one field?

- [Dan] Yeah,

and then if you need to
you can split them out.

- [Matt] And then the taxonomy field

so this is where you get that flexibility

of being able to really

get as granular as you
want with your facets,

so going through your taxonomy terms,

and allowing those to be searchable.

So one of the big problems with this

is that when you're using multiple sites

You can probably imagine you
have a ton of vocabularies

with a ton of different terms,

there's gonna be some overlap there,

and similar to where I showed
you the content type field

you're not gonna want that overlap

where you're gonna have maybe like

two terms in the facets

that have essentially the same meaning,

but just because of how
the taxonomies are set up

you'll have maybe vegetarian
and then vegetarian foods,

where there's not a great user experience.

So we came up with a pretty
good solution to that,

but it takes a little bit more legwork

than the token configuration
that you've seen so far.

So if you tried to edit
the federated terms section

on the search API config

you'll get hit with this big block

that pretty much explains
what I explained to you,

you can't really do this on
the config side of things

you're gonna actually go
into the taxonomy term config

for the settings for that taxonomy term

you're gonna add a field
called confederated terms,

and then you're gonna
configure it term by term

as opposed to having to
take your entire site

and consolidate your taxonomy terms,

or take all of your sites
and consolidate those terms.

This allows you to go in and
do that in a different way.

So this is the interface you're gonna get,

so this is a taxonomy
term called dermatology.

So let's say you have this term

and then you have another term,

or dermatology department or something,

just a derivative of this.

You'll be able to go into
this federated terms field,

and then we're gonna add
a specialty facet option,

so specialty will be the
title of that facets section

and then dermatology will be the checkbox,

and then you can also add multiple,

so as you can see we also added skincare.

So any time someone is on your search app

and they check dermatology
you'll see this,

and if they check skincare
you will also see this.

So it's essentially
building a new taxonomy,

but without having to create the taxonomy

added to all of your content types

and then go edit every single node.

You can just do it in
the taxonomy term itself,

which expedites things a bit.

So this is the interface,

you have the top level which is specialty,

which is that listing

and then you can just
keep going down as needed.

Using the carat symbol.

So Drupal provides,

so putting it all together,

we've said this a bunch now,

but just to recap, Drupal
provides the indexing,

the search page call back that
actually renders that React,

and the search block that
might be in your heading.

So that rounds up all three sections,

so it's kind of hard to follow,

it's a lot easier if you get
into our demo environment

and play with it yourself,

you'll see how you can go through it,

the federated terms
editing those taxonomies

is actually really cool
and pretty intuitive,

and it's a much quicker
solution than the alternative,

so just being able to go
in there and play with it

would definitely be ideal.

So how would you do that,
how do you get started?

If you did want to go
mess around with this,

or maybe add it to one of your projects

Dan is gonna explain a bit about that.

- [Dan] So some of the
stuff that you have seen

we have tried to decouple all

of the pieces as much as possible,

so a lot of that stuff that
we just went over of remapping

isn't necessarily something

that you would only want to do

if you are using this React
app for displaying the results.

So a lot of those pieces,

I guess I'm jumping ahead,

sorry, I'll come back to that.

But essentially a lot of those pieces

are independent modules in Drupal 8.

I think I'm jumping to this one.

Let me come back to that
other one in a second, sorry.

So there is a search API
federated Solr module,

there is also this API field map.

So this API field map is independent of,

it just provides that
mapping functionality,

and you might want to do
that to consolidate values,

even if you're not
using the full solution.

So I skipped ahead,

but this is the URL for the demo site

and these slides will be posted.

The demo site that we use,

we use Mac Development Environment,

so this is what that's
most likely gonna work on,

it may work as well with other tools.

It's pretty simple to set up

once you have some standard tools

like Composer, VirtualBox,
Ansible, and Vagrant setup,

you can spin the whole thing

and is using some task runners
to install those sites,

and this is listed in the repo's README.

To install those sites set
up the example content,

set up the Solr server,

it spins it all up for you.

I think the newer versions are also gonna

configure some of those taxonomy terms,

so even give you a headstart

to see what that might look like.

So again, we talked about
the demo a little bit,

and the demo we are running
is a version of that,

and it uses Drupal 7 and 8.

The Drupal 8 and Drupal 7 modules

are pretty similar,

the main difference is that there

is a little bit more
of a focus on Drupal 8,

just because I don't know,

I think a lot of people
are focusing on Drupal 8,

so that one functionality
for mapping different strings

or mapping the terms is supported

in Drupal 8 as its own module,

but within seven it's just
bundled with the other module.

There are some specific versions,

I don't think there's much
that is important about those,

that's what's part of
the demo environment.

The key point is that it
is using Apache Solr 4.5.1,

what is the version that,

I don't know if 4.5.1
is the exact version,

but it's similar to the
version that Aquis as using

with Aquis search,

and because a lot of people

are using Aquis and Aquis search,

we wanted to test against that.

But the live versions that
we have also tested against

are running newer versions of Solr,

and they do support more features

but there's not a huge difference

in terms of using that version or another.

I don't think you will
have to change anything

unless you wanted features

that weren't supported in
the older versions of Solr.

Yeah I think we will open up questions,

I think maybe, yeah, that sounds good.

So thanks everyone for doing this,

and now we want to open
it up again to questions.

- [Man] I had a question,

so I noticed that the site
was in the faceted search,

this came up actually not too
long ago, maybe last week,

we were looking at our own faceted search

which was a third party tool,

and we were trying to see
if you could do a search

that would only show one site by default,

depending on what site you were coming

from to do the search,

but our search engine
lives somewhere else,

so it sort of hard to get past that.

Does yours have any capability
for that sort of thing?

- [Dan] So the question was,

is there capability for
filtering out to just one site

depending on what site you're coming from.

So there was a release of a
lot of this work yesterday,

and it introduced a bunch of features

that we weren't showing in the demo.

One of the features is support
for more URL parameters,

so some of those are pre-filtering things

based on where you are coming from,

so you might want to filter by that.

So you could have the experience

where you're using the same application

but depending on what site you're in

You either start with it
filtered to a specific site,

or you just don't even show

the option to filter by different sites.

But that was a big request.

I think some of the other
features that were added,

maybe we can talk about those
a little bit real quick.

Was support for searching across

a wider range of those fields,

and adding auto complete,

and adding more wildcard
features to the search.

I think the wildcard maybe

you want to talk a little bit about.

- [Matt] Yeah so if you saw in the demo

initially if there was no search
there were just no results,

and that kind of let things blank,

so now you have the option
just with the checkbox to say

if there are no results
just do a wildcard search

where you would just get whatever

results it wanted to return,

just for a more user-friendly experience.

So you don't have that empty page

when your users first get there,

it might look a little broken.

Yeah?

- [Man] So just to make sure I understand,

the Drupal site,

the modules that you have
running on the Drupal site

set up the React app, set
all the settings down,

and then from there does

the React app fall directly
to the Solr server?

So Drupal is only
involved in counting Solr,

hey, look at these,

and then on the page this is
how you run your React app,

and then the React app then, okay.

- [Dan] So yeah.

- [Matt] Do you remember
where the config is?

- [Dan] It's in search and then (mumbles).

- [Matt] That makes sense.

Yeah?

- [Man] Would you recommend

this for only bringing
multiple sites together,

or are there other cases where you

would recommend this set up?

- [Matt] I think bringing
multiple sites together

is just a feature that
is useful in some cases,

but everything else other than I guess

deciding what sites you
want to see results from

would still be incredibly
useful on a single site

to get your search up and
running pretty painlessly.

- [Dan] The question was if
we would recommend using this

if there was just a single site, so yeah?

- [Man] Did you have to index
any documents like PDFs,

or has it only been web
results that you have done,

I have a number of clients with that need.

- [Dan] The question was have we

indexed documents other
than files essentially.

I don't believe so.

So the only files are
images which are references,

so I imagine if you wanted to extend it

and have an initial field or something

that listed out these are

the URLs for this files or something,

but that's not a feature
that's there right now.

This was what the configuration

looks like on the Drupal site,

so this is just login.

You can set up the app path,

and this is a slightly
older version as well.

But you can configure the app path,

you can add some basic
configuration for titles,

search from the list of indexes,

you can have multiple indexes.

The security features
that we mentioned before,

the SearchDex supports
basic authentication

as a way of protecting
against sending content,

so that's somewhere that
you could fill this in.

And then depending on the
features that you have

there are some options
around the site names,

and you could configure
some other basic features.

So this is basically
if you're gonna display

it on a Drupal 8 site,

you don't have to provide
that much information,

and there is defaults for you,

and then you get that basic page,

and if you want you can
start to customize the pieces

and do things like the styling as well,

so each site has its
own styling and stuff.

Any other questions?

- [Man] What's the scale of the
implementation you guys did,

(mumbles).

- [Matt] I think we had
on the scale of tens

of thousands of pages
that were getting indexed.

I don't know how much,

I don't know what the usage is.

- [Dan] The question was what's the scale

of the solutions that we have used so far.

And we're not sure how
many hits the system gets,

but I think you said tens of thousands

of documents are being
indexed to the Solr server.

I think the limitation here,

the React app is gonna
be served by Varnish

or whatever sort of
caching system you have,

so the limitation is gonna
be on your Solr server,

because it's hitting the
Solr server directly.

So if the solution can scale,

then that's probably something
that would work well.

We have these sites running,

so if you wanted to see some

of the other configuration
interfaces we can do that,

if there's any other questions

we are also gonna be here
walking around the rest of today.

So, come let us know.

- [Man] Have you tried to
use any other CMS for this,

like to send the index to Solr?

- [Dan] Yeah, the
question was have we used

any other CMS's to send content,

I don't believe that we have.

- [Matt] Yeah, as stated before,

it's just a matter of setting that,

whatever CMS you are using,

it's a matter of setting it up

so that the indexing works properly.

So we just happened to
make the Drupal app,

because we are Drupal developers,

or sorry the Drupal module
that does all that work for us,

so it would be pretty extensive

to test it on another CMS by essentially

writing the exact same thing,

but it should work if you
configure it properly.

- [Dan] Yeah and I think the requests,

when you index content in Solr,

Drupal does a lot behind the scenes

to format the connections,

but the submissions are not that complex,

so I imagine it wouldn't be that difficult

especially if you have a specific format

that you were looking to create.

Yeah?

- [Man] Do you know can Solr
handle authenticated content.

So if you have content that's
only for authenticated users,

can you only return those results

for users that are authenticated?

- [Dan] Yeah, so the question is

can you serve different
results to authenticated users

versus in-authenticated,

so right now that's not a feature.

I imagine you could set up,

but right now that's not a feature.

- [Man] So I suppose if
you want to make it easy

for WordPress to adopt the same thing

you would have to implement
as much as you could

using PHP libraries
included with Composer,

and then at your actual Drupal
module would be much smaller.

- [Dan] Yeah, so Sanjay was recommending

using standard PHP libraries
to do the indexing part.

I don't know the specifics
of what the Drupal,

so a lot of this from the Drupal code

is focused around the
existing API modules,

and the Solr integrations,

so I don't know enough

about those to know what they're using,

I imagine a lot of it
is very Drupal specific.

But yeah, if we wanted
to be across more things

that would be a good direction.

Cool, any other questions,

we have some thank you slides

that I want to get to before the end.

Let's do those and then if you
have questions that come up

let us know.

So here's a list of people,

Avi's in this room here.

Adrian is in the room, you
can talk to him as well.

We mentioned Ken.

So there's a long list of
people that have been involved,

University of Michigan
obviously is a big one

that we worked with for
the initial implementation.

For the version two
that came out yesterday,

we're working,

I don't know if we talked about them yet,

but there are people
that we are working with

that are helping support that,

so we are thankful for them as well.

So it's a lot of people,

we are getting some contributions

from outside Palantir as well,

and all that stuff is
available on open source,

so if you're interested in it let us know.

Or take a look at that demo site

and see what looks good.

So yeah.

These are our pictures.

- [Man] Oh, that's what you look like.

- [Dan] Yeah, and then before we end,

if you liked this talk

or if you didn't like this
talk please go to this number,

if you can't remember this number,

you can go to the schedule
page and click on the session,

that's what that goes
to, and leave feedback.

And there is a contribution day

that's on Saturday from 10 to four,

and I think there's more
information on the website as well,

but yeah, that's some standard things.

So thanks everyone,

and if you do have questions let us know.

(audience applauds)

Captions made possible by ClarityPartners.com. Chicago area Drupal consultants