Azure Search Primer with Eric Bowden

Danny Ryan

Host – Danny Ryan

Bio – LinkedIn – Twitter

Sam Marshall

Guest – Eric Bowden

Bio – LinkedIn

Key Points

Four Pillars of Azure Search:

  1. Create a Search Service
  2. Create indexes and fields
  3. Push content into the index via the API
  4. Search the index via the API

Conversation Highlights

  • Adding Search Ability  – 2:36
  • Security and Security Trimming  – 8:59
  • How to start with Azure Search – 13:22
Danny :Hello and welcome to The Two Bald Brothers and a Microphone podcast. This is one of the bald brothers, Danny. And I’m here with someone from ThreeWill. I’m here with Eric Bowden. How are you doing, Eric?


Eric:Oh, I’m doing fantastic, Danny. How are you today?


Danny :I’m doing great. It’s a Friday afternoon. All is well. I can’t complain. It was a great week, and so just looking forward to the weekend. I’ve got a date tomorrow night. Amy and I are gonna go back down to Buckhead and go where we used to live, and go wander around. Go do a little bit of shopping. I’m looking forward to that. How about you?


Eric:All right. Time in the old neighborhood.


Danny :Yes.


Eric:Oh, it’s gonna be a weekend of rest and relaxation.


Danny :I love it. I love it, I love it, I love it. What I wanted to talk to you about is Azure Search. Now, we’ve done plenty of content on the website about SharePoint Search. I’d like to understand more about what you’re doing. Why did you start wanting to learn about this? Let’s get started with just sort of what prompted going and learning more about Azure Search.


Eric:Sure, sure. Yeah. You know, of course, Search is a great feature of SharePoint. It’s a strong feature. The Fast search engine became part of SharePoint in 2010. You know, there’s a lot that can be built on Search and SharePoint. We’ve had projects which are really search-focused projects in SharePoint. That is the only exposure that I really have to the fast search engine. I understand that it is a technology that precedes its usage in SharePoint, but it’s a well-respected feature in SharePoint. Then I’m hearing, okay there is the fast search engine. There is Azure Search, which is a lot like the search that you’re accustomed to seeing and working with in SharePoint. And that really peaked my interest. I’m saying, “Well what” … SharePoint Search already has a great reputation with me. It’s a highly used feature, but what is this Azure Search? I’m hearing it’s a lot like SharePoint and I want to find out more about it.


Danny :Awesome. Awesome. How did you get started with learning more about it?


Eric:With my favorite go-to learning source, which is Pluralsight.


Danny :Awesome.


Eric:I head right into Pluralsight and found a great course called Adding Search abilities to Your Apps with Azure Search by Chad Campbell. This was a great course. Provided a really good high-level and a deep understanding of how I can get started with Search in Azure.


Danny :Cool. Let’s dive into it. Let’s maybe start at the 25,000 foot level. We’ll call it that. What were some of the things to get started with Azure Search? What were some of the things that you picked up?


Eric:Sure. Yeah. The good news is that, for one thing, if you are familiar with SharePoint Search, Azure Search will be equally familiar. There’s some translation of terms. Some things, you know, seem to have been re-named, for example in Azure, they use the word score, whereas I’m more used to hearing the word ranking in SharePoint. They use facets in Azure, where they use ranking. I mean, excuse me, where they use refiners in SharePoint, so there’s some translation to occur. But in large part, it’s very similar to SharePoint.


Now, I start off by sort of building, you know there are really four pillars of knowledge with Azure Search. The first is there is, you’re gonna create a search service in Azure. You go in, create a search service. It’s actually a fairly minimal set of inputs to create a search service. You’re gonna give it a name. You’re gonna give it a resource group and a choose the pricing, the sizing of your Azure Search. A few other fields, but it’s fairly minimal. There’s your search service, pillar one.


Pillar two is you’re gonna create indexes, which in SharePoint we’re not used to creating indexes. In SharePoint, you have one index, but in Azure Search you could create more than one. Then from there, you’re creating fields for that index. This is equivalent, if you kind of translate over into the SharePoint world, this is the equivalent of configuring managed properties, or crawled properties and managed properties. It’s really both of those you’re doing in Azure Search. You’re configuring your fields. At that point, you have a search service and you have an index.


Then the third step is you’re gonna need to get content into your index. That’s accomplished with the API. There’s an API call. You’re gonna push content up into Azure. There is also, there are indexers. There’s indexers capability, which on the SharePoint world you would know that as a crawler. Then finally, that leads me to the fourth pillar, which is searching the index. You’re gonna execute API queries against the Azure Search service to find your results. And of course, all four of those topics are very rich. They’re, certainly indexes and searching are very, very rich and you can go very deep in what Azure provides for you. I think that a really good key takeaway for somebody who is familiar with SharePoint Search and are thinking, you know, “What is this Azure Search thing and what are the complexities and really how can I get into it?” Just keep in mind those four sort of pillars of knowledge there.


Danny :Awesome. Are you indexing SharePoint? Is it just a website that you’ve created or an app? What’s some of the typical things for you to index?


Eric:It can pretty much be anything. I am at the beginner, the intro level of knowledge myself with this, but you know, the way that I see Azure Search being used versus SharePoint Search is Azure Search is I think gonna be appropriate for these sort of isolated solutions. It’s a solution that has an isolated set of data. It’s not, you know, part of your internet. It’s not part of an aggregate. You know, your enterprise fortress of content. You can push any record, so it can be documents or it can be records with fields. Then also, of course, you can configure the indexers that I talked about, so you can crawl content sources. It can crawl sequel databases, cosmos database, blog storage, a couple of other content sources. It’s a variety of content that you can push into Azure, into the index.


Danny :Then, I imagine you have some way of scheduling when the crawls occur and whether they’re incremental or full crawls, and those types of things?


Eric:Yes, exactly. Very similar process, as you’re familiar with in SharePoint. You can schedule the indexers, you know when those are gonna crawl over your content sources.


Danny :I wonder, how does this fit into … I guess if somebody is using this Azure service and they’re indexing content, and I guess for customers who might be, let’s take SharePoint on prem, and they’re on prem for a certain reason, where they want to keep that data within their environments. This wouldn’t be a solution they could go with because it’s you’re actually pulling that content up into Azure. Am I reading that right?


Eric:Yep, you’re right. Yes. It’s a Cloud service and you are pulling. What you’re pulling into the Cloud are those elements that you want to search. You know, which most likely, if you have concerns about data and you’re keeping that data on prem, most likely the aspects of it that you want to search are also sensitive. But it is, yeah it’s a Cloud hosted solution in Azure.


Danny :How does security and security trimming fit into this?


Eric:That is a good question and something that I haven’t fully explored yet. I was actually discussing that with the team yesterday. You know, there are two aspects of security to keep in mind with Azure Search. The first is your access to the services. Access to the service itself is covered using access keys. A number of services, Azure functions is one that I’m familiar with in Azure, that support the idea of access keys. Essentially it’s a course access mechanism, where you can create these access keys, which simply give you access, you know, to call those services. But then the next step is gonna be saying, “Well, what about the specific items in the index? How are the specific items security trimmed?” Folks that are accustomed to SharePoint, we’re accustomed to, you know the SharePoint Search index being security trimmed. You execute a search in SharePoint, you don’t see results back for content that you don’t have access to. I don’t know what that story is yet in Azure. You know, I would imagine that, I think you’re on your own, but I don’t know that for sure.


Danny :Okay.


Eric:You know, where you’re gonna need to be coding in either a mechanism to store the access control list that’s part of the search index, so that’s sort of a front-loading method. Or the other side is, you know, when you retrieve those contents, you’ll have to have a mechanism to decipher if the users have access to it.


Danny :Any projects that’s from your past, now that you’re starting to learn about this, where you see where you might have used this on the project? Or maybe taken a closer look at this?


Eric:Well, sure. I mean, a recent project was a knowledge-based solution built on SharePoint. It was essentially a search-focused project. A lot of filtering, refiners. A lot of logic. We were doing promoting of results. Increasing the ranking of specific results and doing some tuning, and so forth. This potentially could have been a good candidate for that, and there are a couple of reasons for that. One reason is that the solution didn’t have deep security requirements, security trimming requirements.


Danny :Gotcha.


Eric:You know, we needed to secure content in buckets. For this project, they were by client. It wasn’t down to the individual level. You know, the security trimming aspects wouldn’t have been an issue. But then also, you know, this solution, the knowledge-based solution, it was a very specific built application for this knowledge-based purpose, for a mission critical purpose at this company for our client. You know, looking at building it on SharePoint, SharePoint is more of an enterprise-wide internet solution. This might have been a better fit in Azure because it kind of takes what is this purpose built mission critical application and kind of puts it into its own isolated environment, where you can really tune things and you can make configuration changes that only affect this application. It has no possibility of affecting the rest of your SharePoint environment.


Danny :Gotcha. Anything else as far as getting started with Azure Search? I mean, you mentioned Pluralsight as being a great resource. Anything else as you’ve started to take a look at this?


Eric:You know, I just recommend just dive in. When you get there, there is a, of the pricing tiers, there’s a free pricing tier for developers. As you’ll see from the Pluralsight course, and there are other walk-throughs that you can find. Postman is a great resource for executing those search queries.


Danny :Gotcha. Wouldn’t the search … What are you writing search queries in? I mean, is it a certain … What is that?


Eric:It’s actually, it’s called the Lucene search syntax. Well let me take a step back. There are actually two.


Danny :Okay.


Eric:There’s what they call, there’s a simple query language and there’s a full text, a full query language. Which to kind of translate those onto the SharePoint side, you know you might be familiar with Keyword Query syntax or Keyword Queries. And then there’s Full-Text Query. That’s what we had in SharePoint. Then, you know, in Azure we have what they call Simple and they call Full. That Full syntax is using what they call the Lucene Search Syntax, which Lucene is a search engine that goes back a number of years that has eventually been ported into, I guess into dot net. Then we’re able to access it through Azure.


Danny :Lucene, was that like an open … I thought I’ve heard of that in the past, like an open source search engine. I could be wrong.


Eric:Right, right. Yeah. It was actually, it started, I think in the late ’90’s, ’99 or so. I think was originally a Java-based project, and then it was ultimately open sourced and ported to a number of, probably half a dozen or so, platforms.


Danny :I thought it was originally a show – like Lucene and Shirley, right?


Eric:Exactly. Well, it’s pretty funny. I was even taking another direction. I was looking up … I was trying to understand the history of it myself, and I misspelled it. And there is apparently a protein called lucien, if you spell it a little bit differently. And believe it or not, there is a search engine out there for the protein called Lucien. I don’t know if they use that.


Danny :Nice, nice, nice. Well I think this was great. This is great for getting us started off. I appreciate you taking the time to do this, sir.




Danny :Absolutely. Thanks, everybody, for listening. Have a wonderful day. Hopefully good luck with learning more about Azure Search. And have a wonderful day. Thank you so much. Bye bye.


Additional Credits

Podcast Producer – Oliver Penegar
Intro/Outro Music – Daniel Bassett

read more
empty.authorAzure Search Primer with Eric Bowden

Find Anything in SharePoint with Amazon-Like Faceted Search Webinar

Bo is a Principal Consultant for ThreeWill. He has 18 years of full lifecycle software development experience.




Hello everyone, and welcome to this webinar on finding anything in SharePoint with Amazon-Like faceted search, wow I got that out.


This is Danny Ryan, I’ll be your host for today, and I am the co-founder and VP of Business Development for ThreeWill. I have here with me a partner, Ross Leher, he’s the CEO and chairman of WAND INC. I’ve also included in on this conversation as well Bo George. Bo is a principal consultant for ThreeWill. He leads up our portals practice lead. I wanted him in on the conversation as well.


The way this is set up today is, it’s set up as a conversation. I wanted to have Bo just be able to chime in, maybe on some recent project experience, or some things that we’ve seen, and have him chime in that way, and look forward to showing this to you guys today.


Thank you everybody for joining. A couple of logistical things, before we jump into the content … we will be recording the webinar, so you can share it with colleagues. We’ll send a link out to it, so look for a followup next week, with the link. I think we might also be putting this on the podcast as well.


The deck for the webinar is available, it’s in the downloads panel … let’s make sure we’re all set there. You should be able to see … actually, it’s in the handouts. You can see in the handouts panel, you can download it from there, a PDF of the presentation.


If you’ve got any questions at all, I’ve got our producer here, Oliver is behind me. Today’s Oliver’s birthday, it’s his 21st birthday, happy birthday Oliver. He’ll be handling your questions, or if you have anything you’d like to chat about as well, he’ll keep an eye on that.


You’re not drinking yet, are you? You’re 21 years old … not til tonight. Okay, he’s going to wait til tonight.


So if you’ve got anything there, just let us know, and we’ll respond to the questions towards the end of the webinar. So that gets us kicked off with a little bit of logistics.


The agenda for today … what we wanted to start off with is our perspective and the backstory on this, which is why we wanted to share more information about what Ross and his company does, and why we think this is so important to our customers and to people in general who are using SharePoint and things. So I wanted to give a little bit of a backstory to get us started out with the webinar today. The next part, I’ll let Ross take over, and he’ll help define for us what the enterprise need is, around findability. I also want him to give a little bit of a background on sort of how WAND fits into the picture, what they do. And then, for some people this might be a new topic for them, as far as what taxonomy is, so I just want to give sort of an overview of what that is, and really just be able to make sure that we’re all using the same vocabulary when we’re talking. And then, sort of looking at what this overall webinar’s about, we’re looking at trying to create something, like Amazon-like faceted search.


We all know, the net time you go look for a new pair of shoes within Amazon, it’s amazing how quickly you can narrow it down to the correct ones. So we wanted to say, “How can we do something similar with the content that we have on our intranets, inside of SharePoint?” And then he’s going to go through and just share a little bit about the business functions in industries, the different taxonomies for those, and then I’ll ask him to do a demonstration of their taxonomy portal.


And then from there, sort of like, “How do you get started with all this stuff?” There’s a two-to-three week quick-start project, which we’ll talk a little bit about, and what does that look like, and how do you get this whole thing started out in the right way?


And then we’ve got a case study that we’d like to review about Goodwill industries, and review what some of the things were that we found out from that case study.


Then we’ll wrap it up with, I’ve just got some conclusions, just some things, what do I want everybody to walk away with from this webinar, and just some of the conclusions. At that point in time, we’ll go through and see if there’s any questions that folks have, and we’ll answer the questions at that time.


I’ve got it open right now. I like sort of conversational. I’ll hand it over, after I’m done with the backstory, over to Ross. But Ross or Bo, feel free to jump in, if there’s anything that you’d like to add to the conversation.


Hey Bo, how’s it going?


Bo:Good, good. How are you, Danny?


Danny:Good, good, good. And Ross, you’re there?


Ross:We are here in the mile-high city of Denver, Colorado. It’s a beautiful day.


Danny:It is? Okay, awesome. Great to have you guys here.


Let’s get started with the backstory. For folks who are on the line, I ended up inviting a lot of people to this who are current client, or folks that we have interacted with through the years. Most people know we’ve been working with clients on building intranets and extranets on SharePoint for a while now. It’s been a while.


One of the most common complaints that we hear from people is, “I can’t find anything in SharePoint.” Bo, you hear that quite a bit?


Bo:Yeah, for sure. I was thinking, as soon as you have two people working on a single document, you’re going to have the, “I can’t find anything in SharePoint.” That’s how simple it is, is two people working on something in SharePoint.


Danny:Never mind an organization with thousands or hundreds of thousands of people.




Danny:So the taxonomy is going to enable ways for us to present and find content. Bo, I know you wanted to specifically call out these things: search, navigation, automatic tagging, and document routing.


Bo:Yeah. When people talk about, “findability,” immediately you gravitate towards, “search,” and that’s kind of the most obvious thing … conversational search, your Amazon scenario, is clicking things and refining your results. So that one’s a huge one with taxonomy. The other thing that I think taxonomy enables in SharePoint that people don’t think about as much is navigation, which could be search, or it could be other things. And then automatic tagging, so where I put a document, it could be automatically tagged with terms and taxonomy stuff, which reduces the need for people to fill out metadata, because it’s done for them, auto-magically.


And even document routing features in SharePoint, which I think are awesome, like … I upload it, I tag it with something, and it puts it in specific places, so that it’s organized based on how I tag it.


So it’s not just search, but that’s a huge part, but it also enables all kinds of other features in SharePoint.


Danny:And all this is going to be missing if you don’t have a common taxonomy that’s been defined.


Bo:Exactly, yep.


Danny:You seem as though this is a daunting task, ’cause this is not typically something that we learned in school, or people taking this on. I just wanted to mention, I know a recent example where you were working with a client and asking for, “Give us your top 20 document types,” and even that was a difficult thing for them to go through.


Bo:Yeah, and it’s been my experience, not many of us have graduate school with a degree in Information Sciences, that talk about classification systems and things like that, that’s just not our day job. So when you say, “Give me your top 10 document types,” on the surface it sounds like an easy question, but it may not be.


Some customers will just punt on it and not do anything, which is maybe the worst thing to do. The other flip side is, I’ve worked with customers that, they’ve taken on that daunting task, and it’s taken them years, potentially, to define a taxonomy, because they started with a blank slate and di all that work.


So it can be daunting to the point where you just don’t do anything, or it just takes forever. I’ve seen both sides for sure.


Danny:And Ross, not everyone’s like you, where they eat taxonomy for breakfast.


Ross:Well, it’s not that difficult, but we’ll have some fun today. Look forward to it.


Danny:Awesome, appreciate it.


So starting with what we found, and this is a ley thing from meeting up with Ross, is really understanding … if we can start with a pre-defined taxonomy, a lot of … you just do this in general, consulting-wise, instead of starting with a blank slate, having something that is in initial skeleton for you to work with, or a straw-man that’s out there, it’s really a way to make this a reality.


This is why we wanted to … we didn’t have this webinar initially scheduled for this year, but as soon as Bo and I met up with Ross a couple times, I was like, “We’ve really got to share this with folks in the ThreeWill community.” That sort of brings us to where we are today, and why we wanted to share what Ross has for you.


Ross, I’m going to go ahead and switch over and make you the presenter. I’ll give you control, and if you want to turn on your webcam, that’s fine as well. It’s up to you.


Ross:Sounds good, I hope I’m not dominating the screen too much.


Danny:No, you look great.


Ross:Well, thank you.


Danny:You don’t look that great. Wait a minute.


Ross:Listen, first things first. I want to thank Danny and his team for organizing everything today. It’s terrific. Hopefully we’ll have a little bit of fun today.


The Enterprise Need – The Pain Point. What the heck are taxonomies, and what does it mean to all of us? The problem, I think as Danny just talked about, is search. We can’t find stuff. If we think about the physical assets of an enterprise, we have inventory control systems for all of our physical assets, we know where every component is, if we’ve got a manufacturing line … all of the physical assets, they’re on our ballot sheet, real estate, equipment, so on and so forth.


All the physical assets of an enterprise have been tagged and classified. Why? So we can find them. The macro-problem, 75% … and this is probably understated, 75% of information workers say that finding the right information, not the physical assets, but the intellectual, the information assets, is critical to the organization’s success. We would suggest taxonomies as one of the key success factors for effective search and findability.


What are the obstacles to finding our information, our intellectual assets well, everyone knows poor search functionality, inconsistency in how we tag content. Lack of adequate tags. We waste a lot of time when we can’t find the documents we’re looking for, and we get very, very frustrated. When we can’t find the documents that we’re looking for, we either do without them, or we try and recreate them. But in every case, it’s extraordinarily frustrating. So how do we solve this problem?


Little bit of background about our company first. WAND has been building taxonomies since 1983. We began to build taxonomies a long time ago, because we had large insurance companies here in the United States that we provided software programs to, that required taxonomies. Taxonomy libraries that we’re going to look at today, there’s 128,000 curated terms, address virtually every industry vertical and all of the operational areas of an organization. For some of our clients, we’re necessary. We provide taxonomy professional services. Our client base all over the world, taxonomies can be used in SharePoint, but they can be used in big data applications, eCommerce cataloging … search, as we’ve discuss, BI analytics, the list goes on and on and on.


Out of the box, the taxonomy term store in SharePoint is empty. There’s nothing in it. WAND provides taxonomies that are formatted for direct input into that empty term store. But the enterprise may have many, many applications into which taxonomies can be imported, BI analytics, big data, the list goes on and on and on. We format the taxonomies for direct import into those various applications.


Taxonomy is a mystery for most people. I think most of the folks on this call know what they are. I hope I do, but we all face the burden of explaining to people in our companies what the heck that word, “taxonomies,” means, and what difference it makes to what we’re doing.


Say, for the sake of argument, we’ve got a company that’s got 2,000 employees. I’m going to put those 2,000 employees into a room, and I’m going to ask them the question, “Everybody that knows what a taxonomy is, raise your hand.” Not too many hands go up.


Then I’m going to say to those folks, “Everybody that’s used, raise your hand.” Now everybody raises their hand.


I’m going to say to the people, “When you do a search for the word, ‘shoes,’ on Amazon, you’re going to get 550,000 results. On the left-hand side of the screen, are taxonomies. A taxonomy of size, I want a size 11. Taxonomy of colors, I choose black. Type of shoe, I select men’s dress shoe. Taxonomy of brands, I choose Allen Edmonds. 550,000 results, narrowed down to the five that are relevant to the user.”


I say to those 2,000 people, “As long as you’ve been using Amazon, you’ve been using taxonomy. You’ve just never put that word to it before. It didn’t matter whether you were searching for a pair of shoes, a computer, a television … the taxonomy’s always on the left-hand side of the screen, to enable you to find the results that are relevant for you.”


I’d say to the folks, “Think about the feeling of control we have over our search when we use Amazon. We know we’re going to find what we’re looking for, because those taxonomies are there to help us navigate the result sets. Imagine having the same feeling of control over the search for our enterprise documents.” This is what the WAND taxonomies, in conjunction with SharePoint, enable.


The analogy is a very good one, and if you think about the problem that Amazon had when they started, they had stacks of shoes, stacks of computers, libraries of televisions, so on and so forth, and they needed taxonomies to be able to tag those various libraries of merchandise to enable that taxonomy-driven search.


The enterprise has the same problem. Instead of libraries of shoes, computers, televisions, the enterprise has libraries of HR documents, accounting documents, IT documents. If you’re in the building and construction business, building and construction documents, so on and so forth. And WAND provides taxonomies to tag those various libraries of documents.


The analogy is another good one if we think about a content type. We might think that, “shoes,” is a content type. If we have a library of HR documents, let’s say they’re compensation documents. We may say, “HR compensation documents.” That’s a content type.


So, remember that analogy. It explains everything.


One slide we like to show folks, you can show this to the business side, the technical side, and people immediately get it. Everybody knows what the Amazon experience looks like. Do a search, you’ve got refiners … let’s see what this actually looks like.


In this case, Amazon decide they’re going to have a library of shoes. What we decided is, we’re going to have a library of HR documents. A library of documents, nothing confidential, just documents, insurance plans, company handbooks, time off forms, so on and so forth. And when we check documents into that library, we tag them with the appropriate taxonomy terms. The same terms that we use to tag the documents, navigation trees, in the search process.


When we do a search for the term, “HR,” in that library, you can imagine we’re going to get everything on the left-hand side of the screen, just like Amazon. There’s the refiners … and I need an expense form. Instead of calling my manager, instead of calling HR, I can now find all of the documents that are relevant, open-source documents for employees.


The reason we always like to start with a library like this is because it’s a success story, and it’s relevant to every individual in your company. We like to break taxonomies down even further to their absolute basic levels.


We would suggest taxonomies are the data model for unstructured text. Now I’m going to say to everyone, there’s no such thing as unstructured text. If I put a ThreeWill brochure on the table, if I put a Wall Street Journal article on the desk, everybody’s saying to me, “Ross, two perfect examples of unstructured text, what ar you talking about?”


And I’d say to everybody on the call, “Every one of us can read the ThreeWill brochure, we understand it perfectly. We can all read the Wall Street Journal article. It’s not unstructured text for the human brain.”


So make a small modification to this first statement. Taxonomies are the data model for unstructured text that we can use in computer programs like SharePoint, to tag documents in a structure that replicates how the human brain organizes the domain of information.


We would suggest implicit taxonomies are how the human brain organizes information. These implicit taxonomies have been developed ever since the day we were born.


I have a five year old granddaughter. The first terms she knew were the terms, “wet,” and “dry.” The terms, “hungry,” me stomach is full. She didn’t put words to those concepts, but from the day she was born, she started organizing her knowledge base.


She’s five years old. She knows there’s a top-level term of, “food,” a narrower term, “fruit … apples, oranges, bananas … ” Narrower term to food, “Baked goods, cookies, brownies.” She’s organized that domain of information, “food,” into logical, broader, narrower, narrower, narrower terms.


We do the same thing with geography. Continents, countries, states, cities, districts. And we do the same thing with all the enterprise domains of information. The explicit taxonomies that WAND has developed are simply representations of how the human brain and enterprises organize information. These explicit taxonomies are used with enterprise applications like SharePoint for tagging documents, for taxonomy-driven search and libraries, and of many, many other applications for text-to-knowledge purposes.


We want those 2,000 people to say to themselves, “Boy, four or five minutes ago, I had no idea what the heck a taxonomy was. But I’ve been using the all my life. Been using them when I use Amazon. Heck, I’m going to the grocery store after work tonight, there’s taxonomies everywhere … meat department, dairy department, milk, cheese … different types of cheese. There’s no mystery about that word, ‘taxonomies,’ except one … why have I never used it before, to describe how information’s organized?”


The reason I spend time on this, four or five minutes, is because we find, within our clients, when everybody understands, it eliminated the mystery of that word, they see, “Boy, we’re going to a great search, it’s going to be just like Amazon.” It creates a tremendous amount of momentum, and it makes your jobs a lot easier to implement a great search, and to implement these types of processes within SharePoint.


We take every document, we’ll say [inaudible 00:28:09] your telecommunications company. If we take every document in your telecommunications company, we say, “We’re going to put al those documents, we’re going to put them into one room, just like Amazon did. They decided, ‘We’re going to put those into stacks, we’re going to put them into libraries.’ We’re going to have libraries of accounting, environmental, HR, IT, libraries of documents, every organization has, WAND provides taxonomies that address all those libraries of documents.


Also going to have to have libraries of documents that address the business activities of your organization. Telecommunication documents, a lot of telecommunications companies have retail store, sell retail stores. Going to have real estate documents, going to have building and construction documents, every company build and constructs assembly lines, remodels, whatever it might be.


In a nutshell, WAND provides taxonomies that address all of the domains of knowledge for an organization.


With that, we’re going to go to a demonstration of our WAND Taxonomy Library Portal. What this provides is online access to all of our WAND taxonomies, our client can browse, download the taxonomies in formats that can be directly imported into SharePoint. We provide regular updates, release new taxonomies on an ongoing basis, and the library can be licensed for a single application, or it can be licensed for any, many applications that the enterprise might have.


I’m assuming that everyone on this call has an understanding of where taxonomies reside within SharePoint, and that’s the taxonomy term store. We’ll go to the taxonomy library. Each one of our clients gets a username and a passcode. This particular user, Ross Leher, is licensed to download taxonomies that can be imported into these various SharePoint applications. Just for context, we format the taxonomies for about 100 different enterprise applications, BI analytics, big data, the list goes on and on and on and on.


We partner with many of these application providers, probably our highest-profile partner, WAND provides taxonomies that can be imported into the [inaudible 00:30:34] engine applications.


Taxonomies improve the performance of many, many enterprise applications, and they’re a very, very necessary ingredient. What our users are able to do is, go to the taxonomy library and browse the various taxonomy titles.


Because there’s so many of them, we’ve grouped them into industry verticals. We’ll take financial banking organization, investment bank organization … we’ll say to them, “This is a list of all of the domains of knowledge within your organization. You’re going to have real estate documents, libraries of procurement documents, libraries of finance and investment banking documents, customer service … every organization provides health insurance, life insurance have to have casualty insurance, accounting documents, libraries or facilities management.”


These represent all of the libraries, all the domains of knowledge. And always start with IT for the demonstration. I’ll explain to you why in a moment.


User is able to click into any one of the taxonomy titles and explore the various taxonomy trees. Numbers of terms in a taxonomy hierarchy on the right hand side, in parentheses.


We’ll drill down on IT administration. Many of our clients are using the second and the third-level terms as a basis for setting up libraries. I want to set up a library for all of my IT policies, make sure we’ve got them all. When we set that library up, these are the terms that we’ll be able to tag documents with, when they’re checked in. And just like that HR search we were looking at, these are the terms that will be used as the refiners on the search process in SharePoint.


Some of our clients will come back and say, “Ross, we didn’t have a smartphone policy, we didn’t have some of the access control policies,” soon and so forth. We began to checklist, because in a lot of cases, they had a lot of these policies, but they were in this file share, they were over here, they were someplace else, or they hadn’t formalized them, or they hadn’t updated them. So a great byproduct, it became a wonderful checklist to make sure that they did have everything.


The reason I always start with IT, I make the argument, and every day when you read the newspapers, IT security, most important taxonomy for any organization. We don’t have documents, we don’t have workflows … if we don’t have expertise within the organization that we can tag with these terms, we may want to take a second look at our IT security policies.


What the user is able to do is explore the various taxonomy titles. We can then highlight at any level, and whatever we highlight, we can click on Download. Or, if I want to take all of IT administration at one time, I highlight that, click on Download … select the application into which it’s going to be imported. I confirm it.


Starting to add multiple languages select our language. Click on Continue To Download, and what’s taking place now, IT administration is being formatted for SharePoint. It takes three or four minutes for the formatting to take place. The user can continue browsing, or we can go to our delivery page.


We formatted this just yesterday, IT administration formatted for SharePoint. We can download the file directly to our desktop, and then proceed to import that term set directly into the taxonomy term store in SharePoint, and proceed to use it in our SharePoint implementation.


You kind of want to think about, the taxonomy portal represents like a grocery store. Everything’s there, you can pick and choose what you need, and if you want to think about … go back to that Amazon analogy, each one of these represents … shoes, computers and this and that, all the different domains of knowledge, and I may want to have a library for … just like we look at, employee benefits, and the library for company policies.


This is kind of our menu. What libraries do we want to set up? I want to set up an IT policy library, I want to set up a library for all of our IT infrastructure, all of the different components within that infrastructure, the documents, so on and so forth.


Danny:Ross, quick question as we’re looking at this. From your experience, is there a specific executive inside of larger companies that owns the overall taxonomy of the company? I know there’s certain departments, like an HR department where they can sort of define their own. Who’s the person responsible, overall in these large organizations, who’s responsible for defining the taxonomy? Is there typically an executive?


Ross:It’s not a matter of defining the taxonomy per se, but it’s a matter of realizing, recognizing, we have intellectual assets. We have knowledge within the organization. I don’t think it’s the IT department. What we’re seeing more and more, there’s an increasing … frankly, increasing sales, because there’s a recognition of, Chief Information Officer, we’re seeing new officers, Chief Knowledge Officers, Chief Innovation Officers … and they look at the requirement and say, “We’ve got all this information, and no one can find it.” It’s crazy.


Danny:And then they would work with the different departments, or they would work with their industry specialist to go out and define these taxonomies.


Ross:That’s it. What we’re seeing is … and we call these, “foundation taxonomies,” our clients tell us, it gets the 85%, 90% of the way there. Our client are telling us, it literally saves them months and month and months and months. They’re able to customize the HR taxonomy list in a day. Our clients tell us, “It gets us 85%, 90% of the way there.” Some of our clients tell us, “It’s got us 140% of the way there,” because there’s terms and phrases they would have never thought about, had they built it from scratch.


This is the key point: people are expert at what they do, not experts in taxonomy building, editing … the pre-built taxonomies provides an easy, contextual reference.


Danny:It’s almost like when you hear, “I know it when I’ll see it.” Like you have to see it first. And as I mentioned earlier, it’s commonly. From a consulting standpoint, “Never show up with a blank slate.” Show up with something that you can at last work with, and take apart and build up. This provides that to you.


Ross:I’m going to do, cause we’re in the finance area, “I need a taxonomy for risk.” Huh. Risk. Risk can have many, many meanings. The search result on the left-hand side of the screen, the terms and phrases. On the right-hand side of the screen, the taxonomy they’re located in, from a contextual point of view.


I don’t care about insurance, I can are about credit risk in context of banking. I click on that term, and it takes me directly to that term, “credit risk,” in the banking taxonomy.


Some probably need to have a library for risk management processes. And a library for internal process, and so on and so forth. A library for the various bank regulations, so we don’t have to go searching for them, so on and so forth. A library for the different bank products.


Remember, we can tag a document with more than one taxonomy term. We can have a mortgage product, and a brochure about that mortgage product.


I’m going to show you one other search. This is an example or a multi-faceted search. What we’ve one here is, we’ve indexed a number of SharePoint conferences. When we were looking at the HR library, we just had one refiner. Now, when we tagged the documents that went into this library, we tagged them with the type of document, the conference locations, and the event.


And now I want to find document, conferences in Chicago. I don’t care about SharePoint Fest, I care about Microsoft Ignite. And I don’t care about sponsor schedules, I need to get my registration form.


So if we think about this we have a mortgage product … and we have different types of mortgage products, credit cards, so on and so forth. We may want to tag the various products that we have, and because we’ve got a great sales and marketing team, but they can’t find stuff … when we check that document in, it’s about a mortgage, and the document could be a case study, it could be a data sheet, it could be a sales brochure, it could be a testimonial.


When a salesperson is looking, “I’m trying to sell this product to my client … ” Well, here’s the product, here’s all the documents for the product, and here’s the lead-behinds, and here’s the product data sheets, and so on and so forth. It could be the price list, the price list exist in here, so on and so forth.


And the marketing people … I’m focused on marketing mortgages in the Georgia area. I probably want to be able to find my research, demand forecasting. Or maybe I’ve got the international, I want to see demand forecasting for the UK marketing, so on and so forth.


You want to think about, “How do I organize information?” At the grocery store, how do they organize stuff? They’ve got meat in one area …


Think about your house. You’ve got a kitchen. We’ve got different things in the kitchen. We’ve got a place where we put our spices. We’ve got a place in SharePoint where we put our sales collateral documents. It’s the same kind of a concept.


It’s not difficult because we do it every day. We just have not thought about it, in these types of a context.


Danny:What if your house is not that organized? You see how you could benefit from it.


Ross:This is where you have to start. You’ve got to recognize … and I say this to people, if you’ve been living in your house for 20 years … and this is a great time for a migration strategy, and you’re moving. When you move from House A to House B, you make an evaluation on House A, “What don’t I want to move? What have I not used?” And then, “Where do I want to put it in House B?”


I want to be very straightforward. WAND is a company, we provide great taxonomies. But experts like ThreeWill, they provide the How. Our clients, they want to have great search? Why? “Because we’re wasting a lot of time, a lot of frustration.” But every environment is different, every organization is different.


I don’t know how many engagements, hundreds if not thousands of engagements ThreeWill has been involved in, they’ve done it a bunch of times. Hundreds of times, so this is the expertise, and this is how we integrate with folks like Danny and his team.


Danny:A couple of other things Ross. You just mentioned migrations, and that’s probably a good time … when I think about organizing. We’ve been doing the latest, everybody’s moving typically to SharePoint Online, so I think that’s a great time for you to … a lot of customers want to do cleanup at that time, probably reorganizing, and it’s probably really a good time for you to introduce that taxonomy to the company.


Ross:Absolutely. At the end of the day, we want to be able to find stuff.


Danny:The other thing is, I noticed … if you go back to that search page that you were on, is that Microsoft is starting to, based upon things like the document type and the author and created date, they’re trying to implicitly start to create some facets for you. If you look at the bottom of it, it’s almost like there’s certain things that Microsoft can do like the author and modified date. I think this is something relatively new to SharePoint Online.


There’s been some things that have been around for a while, but there are some new stuff. I find this incredibly helpful, to have that modified date, ’cause I’m looking for a document I know I worked on last week, and it’s not even fining things that other people have created, it’s my own stuff. It’s a huge help.


Ross:I think what you want to think about, when we think about metadata or taxonomy is, we could call this, “administrative metadata.” It tells us that it is an Excel document, it tells us who the author is, and it tells us when it was modified. But it doesn’t tell us what the document is about. This is what we would call, “descriptive metadata.”


When we go back to our slide on the shoes, this is the descriptive metadata. It tells us the size, the color. It tells us what’s in that box of shoes. This is telling us, what’s in that Excel document that Danny authored on June 26th. This describes what it is. Or if it’s a specification sheet on XYZ computer that I’m a salesman for. So this is the descriptive metadata versus the so-called administrative metadata.


Danny:And that administrative data, although helpful, it’s definitely helpful in that, along with sort of a free-text search is helpful … it just doesn’t get people to where they really need to be. It’s like one of those things, if you don’t take that last step, the experience is just … commonly what we’re running into, which is, there’s and time spent towards this, and they’re trying to use that administrative metadata, and they’re trying to use what’s inside the content of the document, but it just doesn’t get them to a place where they can easily find things.


Ross:Another aspect of search … and the guys at ThreeWill, they know I’m not a SharePoint specialist. That’s the first thing I tell people. But in the term store, we call these foundation taxonomies, and in your company, you may have a special location, Georgia, rule and regulations. So you can add terms, you can delete terms, so on and so forth.


Another important point is … we may think of the concept, “leave and time off.” Or, HR. I call it HR, Danny calls is Human Resources, Bo calls is Personnel. So, within SharePoint, we can add synonyms. Retirement plans, whatever the synonyms …


Danny:A real word, for reference, is we would call it, “People.” We sort of call our HR, “People.”


Ross:There you go. But this is where the customizations, the adding of additional terms, adding vocabulary that’s used within the organization. Call it Information Technology, call it IT, so on and so forth. You simply add synonyms.


Bo:Ross and Danny, I wanted to jump in on, when you guys were talking about the administrative taxonomy versus the descriptive taxonomy. Another thing that it made me think, where we help a lot is, taking WAND’s taxonomy and, like Ross said, you apply it to a company, and every company is different in terms of what they call stuff. But they’re also different in terms of their people and their people’s willingness to tag things with the descriptive metadata part.


Administrative metadata, I also, internally, call it the, “free metadata,” the stuff that it doesn’t require a used to anything other than upload it. One of those areas when I was talking earlier is, simply by virtue of where you put something, that could give you the ability for more free metadata, i.e., if you uploaded a document to an HR document library, we might automatically tag it with some level of that HR taxonomy, versus if you had uploaded it to an IT one. So your SharePoint site-structure, document libraries and so on can enable getting some free descriptive metadata, I think is what that kind of boils down to.


Ross:I think that’s a great point, Bo. If we think about our houses … again, I use this metaphor, the house. I use this analogy, here’s the spice rack, and in the garage we’ve got our tools. We’ve got wrenches, screwdrivers, so on. It’s a pain to tag things, it’s a real pain when I do some work in the basement on the water heater, to get all the tools back and put them back where they belong. It’s not a lot of fun, but at the end of the day, if I don’t put those tools where they belong, if my wife doesn’t put the spices where … we can’t find them.


When people realize, “This isn’t that difficult, and I’m going to be able to find my stuff, I’m going to be able to find document that Bo checked in, that Danny checked in, we’re all going to be able to find … we’re going to start to eliminate this frustration.” Then, there’s a lot momentum, because it dues alleviate a lot of frustration, and it enables us to do our work a lot better.


Overview For a Successful Managed Metadata Project. We always tell folks, “Get a quick win.” And what we want to be able to say to our organizations, “We’re not going to have deliverable results in two to three years, not two to three months … two to three weeks is what we’re looking for.”


Danny:Now you’re talking.


Ross:And our goal is findability. We want to tag our documents. Why? So we can quickly find them in search. How do we get that started?


And if we go back to this financial organization, the folks do a library on banking, the banking folks are going to be interested in that. The procurement people ae going to be interested in the procurement libraries, the accounting people the accounting libraries. So there’s going to be a limited group of people interested in these various libraries … with that one exception. Every employee at the bank, from the CEO to the latest new hire, wants to quickly be able to find their policy documents, benefit documents … again, nothing confidential.


So, that’s the group we start with. We download the HR taxonomy, get somebody from Human Resources to help customize it, generally that takes three to four hours. You’ve got a limited number of terms there. Takes 30 minutes to configure the SharePoint taxonomy columns and libraries, search refiners. Have that same person from HR assist in the tagging of the documents. Generally, there’s 250, 300 documents of that character, three or four hours to do that, maybe a little longer.


Then we announce to every employee in the company, “Library’s open.” And they see that search works, and they look at this and they say, “My goodness. If search can work for this library, why can’t they do that for my sales documents?” And the IT people the IT documents, and everybody starts to project what this is going to look like in their day-to-day world, within the organization.


It’s not an IT project anymore, it’s not a SharePoint project. It’s a project that everybody sees, “What’s in it for me?” It generates a huge amount of enterprise momentum … and then your problem is going to be, everybody’s going to be tugging on your shirt, prioritizing additional groups for implementation. Deliverable results, two to three weeks, and now we want to replicate it enterprise-wide.


Nice case study, and there’s a link to this … I’ll see if I can find it here. Nielsen Norman selected the 10 best intranets every year, and Goodwill Industries International was selected as one of those. In the real world, we all know this, we want to be able to find our stuff, we want to be able to find our documents, we want to be able to find the things we have in our houses. Search, findability, findability, search … it’s part of what living is all about, and SharePoint’s a great environment for that. We think we’ve got wonderful taxonomies. The folks that organized this, ThreeWill, Danny and his team, they’ve got the expertise to show you how to get it done.


Danny, I’m going to turn it back to you. I think I’ve talked enough.


Danny:That was great Ross, thank you so much. I’m sure everybody got … it’s such a great presentation that you do, and I really wanted to share that with everyone else.


Just to get us wrapped up here, we’ve got about 10 minutes left before the start of the hour. I wanted to just go through some of what I sort of took away as the conclusions, when listening to Ross and to Bo, and talking with Bo about this.


The key component to addressing a findability or search problem is to define a common taxonomy for organizing information inside your company. We have to do that. Taxonomy enables … and I pulled this from Bo, Bo ha pointed this put before the webinar, which was, it provides powerful ways to present and find content, including not just search, it’s also navigation, it’s also automatic tagging and the document routing.


Coming back to the overall webinar and what we were trying to present in this webinar, which was, how do you turn SharePoint and how this Amazon-like faceted search available to you? It’s there so that your users can find a document the same way they find their next pair of new shoes.


As part of that defining that custom taxonomy from scratch, we’ve just found, is a really daunting task for most companies. The companies who do go after it, as Bo mentioned earlier, they might spend a couple of years doing that. So it’s really a daunting task, and from a lot of what we’ve seen, it’s just commonly not done, and all they’re using is the administrative taxonomy.


Starting with a pre-built taxonomy, it’s an effective way to jumpstart this process, and to get to the point where, if you’ve got 75%, 80% of the way there and you’re going there, and you’re culling and you’re adding, and you’re just making it your own, is really the route that we would recommend doing.


Also, one of the things that’s sort of the last point here is, taxonomies do change over time, and one of the neat things that I saw about Ross’s product is, it will also, over time, as industries update and business changes come, you can keep those taxonomies up-to-date, which is really important, because the business changes.


Anything else to add to that, Ross or Bo?


Ross:I think the updates, we have six masters of library information science taxonomists that review each of the taxonomies on a quarterly or semi-annual basis, deepening on the volatility. If we just think about, 20 years ago, the vocabulary that we were using. We weren’t talking about, “online,” we weren’t talking about, “internet.” Al the vocabulary that didn’t exist. Today, machine learning, artificial intelligence, internet of things, so on and so forth, it’s just all new technologies that we keep an eye on, and we add them to our vocabulary sets.


As you can tell, I’m an older guy, and you can think about … when I was going to high school, they had computer card. You think about your kids, the technology you grew up with, and then think about the technology your kids are growing up with. It’s really remarkable, the changes that take pace. Sorry for pontificating. Back to you, Danny.


Danny:That’s great anything else that you’d add, Bo?


Bo:No, the only thing I was thinking is, for those who are on the fence about pursuing a taxonomy, I would say go for it. I think the worst thing you can do is be complacent and do nothing, and somewhere down the road have hundreds or tens of thousands of documents that aren’t tagged, and then you want to do the taxonomy, it’s probably harder. The sooner you start with something, the sooner you’ll get benefits from it.


Danny:One of the great points … in the chat window, I shared one of the neat things that Ross and his company does is, they do make available to everyone a general business taxonomy. It’s sort of a starting place that has general business terms, and a taxonomy for that. So if you wanted to download that there, as an example, I think that’s a great place for people to start as well.


Ross:If I can just expand on that a bit … Microsoft announced SharePoint 2010 in 2010. We developed the genesis of this general business taxonomy, it’s also available on three blog postings on the Microsoft site. We developed it in conjunction with Microsoft, as a partner. What they wanted to do is, they wanted to provide their client base something for their clients to download to put in that empty term tore for this new feature that they just developed. We’ve actually had about 8,500 downloads over the last five, six years. Danny, that’s a great idea. That gives you a start, give you a flavor of what you can do.


Danny:Awesome. Any questions that folks have? And if you’ve been waiting until now, feel free to ask them now. I don’t think … what do you have, Oliver?


Oliver:Can it help fine-tune an existing taxonomy?




Bo:I was going to jump in and answer. I saw the question in the window, too. I don’t know if it would necessarily help you fine-tune an existing taxonomy. I think it might show you a general taxonomy for your particular needs and user input, but I think the tuning of your taxonomy is probably going to be more closely-aligned with your specific company and needs. But I don’t know what your thoughts are, Ross.


Ross:What I would say is, companies that have developed taxonomies, there’s not very many of them. They generally develop them based upon file plans, the terms that they use for file plans. If we think about a file plan, a file plan is an analog way or organizing information. So, while those exiting taxonomies can be interesting, the formal curated taxonomy that’s designed for digital tagging of documents is much more effective, but you can fold the terms in as synonyms, and that can be helpful in the transition.


Danny:Awesome. Anything else? I think that’s it. I appreciate the question, that was great. It looks like we are getting to the end of the hour here, so again, we’ll send this deck out next week. If you want more information, feel free to follow up with myself or Ross. Here’s our websites.


Ross, thank you so much. It was really informative. I really appreciate you taking the time to do this.


Ross:It’s a pleasure. Thank you, and thanks to all the people that have attended, and to Bo as well.


Danny:Absolutely. Thanks, Bo. Thank you everybody for attending, look for an email next week, and have a wonderful weekend, everyone. Take care.


Ross:Have a great Fourth of July. Bye bye now.


Danny:Absolutely. Bye bye.


read more
Bo GeorgeFind Anything in SharePoint with Amazon-Like Faceted Search Webinar

Office 365 Groups vs Azure AD Security Groups

Caroline Sosebee is a Software Engineer at ThreeWill. She comes to us with 20+ years of software development experience and a broad scope of general IT support skills.

We recently had a client who was ready to streamline the security of their SharePoint Online site and change it from ‘Everyone’ access to groups of people with more specific access. Our recommendation to them was to use Azure AD groups so that the groups would be global and could be both centrally managed and used across site collections.

As we moved ahead with it, they had the groups added with the appropriate members. We then granted SharePoint permissions to the new AD groups by adding them into the appropriate SharePoint groups and removing the reference to ‘Everyone but external users’.

At first all seemed to work ok but as the week progressed, random problems started cropping up that we couldn’t explain, the biggest one revolving around search results. One of their users (and others, we later found out), who had full read access to the root site and all subsites, would only get back results from his OneDrive library and from the separate training documents site (which is open to Everyone). Yet he could easily navigate to and access all the document libraries in all the sites.

Thus began my long search on all sorts of things SharePoint search related, trying to figure out what was going on. For some reason, I finally decided to go look at the AD groups themselves with the thought that since roles are assigned to users, maybe the same thing might need to be done for groups. This was a bust of course, but being fairly new to administrating SharePoint Online, I was game for checking all sorts of things I didn’t know about.

Luckily this random check ultimately ended up pointing me to the real problem. It turns out these two new groups were setup as Office 365 Groups instead of security groups. At the time, I didn’t know anything about Office 365 Groups but didn’t really think this could be the problem. I decided to do a little research anyway into what that meant. One of the definitions I found was:

Office 365 Groups is a service that enables teams to come together and get work done by establishing a single team identity (managed in Azure Active Directory) and a single set of permissions across Office 365 apps including Outlook, SharePoint, OneNote, Skype for Business, Planner, Power BI, and Dynamics CRM.

So SharePoint was mentioned in that list of Office 365 apps, right? How could the group type be the problem then? We needed access to SharePoint and it says it does that. What it doesn’t tell you is that it’s mostly referring to access to the team site that is created, specifically for that group, when the group is first created. It does not mean that it will be very usable by other sites.

After more searching and finding very little, I decided it was time to do some of my own testing. First I had a test user added to the Office 365 Group currently in use. After giving the cloud some time to process this change, I signed in and ran a search or two. What I got back was very similar to the user mentioned above. I got little or no results back, even though I had access to everything in the site.

I then created a new Azure AD Security group, added the same test user to it and then granted it the same permissions in the SharePoint site as the Office 365 Group had. After waiting a decent time so I was sure the security change was processed by search, I signed back in and found that the behavior was now entirely different. With the test user as a member of my new security group, I got back tons of results, just as expected. To further verify, I then removed the user from my test security group, waited a bit, ran a search and found I was back to square one. This was pretty solid evidence that the Office 365 Group was the culprit.

Our end solution was to create new Azure AD Security groups, add the correct members, grant them the same access as had been granted to the Office 365 Groups and then remove the access for the Office 365 Groups. This seems to have corrected all the problems the users were experiencing.

I still don’t know a lot about Office 365 Groups and haven’t had time to research much further, but I do like the below snippet (found here) that very succinctly describes each group and what it does.

I’m sure that Office 365 Groups have a place in the Microsoft world, but it is definitely not as a replacement to AD security groups.

As a quick recap, here are the areas that were impacted (at least on this particular site) by using an Office 365 Group instead of a security group:

  • Search – would not return results from the site
  • Starting a workflow – if a user in an O365 group kicked off a workflow, the workflow got hung up with an ‘Access Denied’ error before it ever got far enough along to send out custom errors.
  • Site access – Various users had problems accessing the site, even though they were in the correct group. Was a very random thing as it worked for some and for others, it didn’t.

I hope this helps save someone some time in the future!

read more
Caroline SosebeeOffice 365 Groups vs Azure AD Security Groups

SharePoint Document Generation Webinar

John Underwood is a Technical Evangelist at ThreeWill. He has nearly thirty years of software development experience. He is an experienced technical instructor with superior presentation skills and is proficient in delivering standard curriculum as well as developing and delivering custom curriculum.

Well, good afternoon or morning or evening to all of you, depending on where you’re at. I’m John Underwood and I’m going to be your tour guide today as we go through the SharePoint Document Generation webinar. I’ve had the good fortune of having some of you attend my webinars and classes in the past and if you’ve come back, I thank you very much.

For those of you that are new, just a little bit about me. I’m lucky enough to be working at ThreeWill. My job there is Technical Evangelist and, really, that involves two things. I get to do some engineering chores where I write code and then also get the privilege of doing a lot of presentations, be they webinars or videos or even face-to-face training classes. That’s something that I really enjoy doing.

If you need to contact me for any reason, you see a couple of touch points there. My email, [email protected] Also, if you like, you can follow me on Twitter @jbutweets. You may want to scribble that email address down really for two reasons. Number one, if we were to have any technical problems during the presentation, you could use that as an alternate way to reach me in a moment of desperation. Beyond that, you may have some questions, some follow-up, et cetera, after the webinar. Always welcome your questions. Always have to give the disclaimer that I can’t act as your off-site help desk, but generally if a customer follows up with a specific question about a topic I’ve taught, I always wind up learning something myself. If you have some questions on today’s material, I would welcome those emails in the future.

Now, some of you may have heard that we were giving away an Xbox today. My little bruised ego, I’m not going to think that you showed up just for that, but on the other hand, if it did motivate you, that’s a good thing. Just letting you know that we’ll give you the details of the giveaway and how that’s all going to happen at the end of the presentation. Obviously, we’d like for you to stay engaged until the very end so that we can share some useful information to you.

Who am I going to be speaking to today? First of all, I’m expecting that some of you are in the role of developer and you may be a developer that’s experienced with SharePoint or maybe you’re kind of new to it. Regardless, I’m going to show you some code today. We’re going to do some things in Visual Studio, but even more than that, I’m going to show you some concepts that will let you go back and help your SharePoint users get the most out of SharePoint.

The second audience that I hope to address today are those of you that just showed up and you’re just SharePoint users and you want to learn some cool new stuff. I think what you’re going to see today is really two things. Number one, some tips and techniques that you can actually go use this afternoon after the webinar is over, and then beyond that, just kind of a knowledge of what’s possible. Sometimes that’s really helpful even if you’re not the person that’s going to write this little application that does this wonderful thing. Just knowing what it possible will then allow you to go back and have a conversation with your developers about what you want to accomplish.

Then, finally, some of you are just kind of more in a leadership or a business decision-maker role and more than anything, through the code and the concepts and the tips and the techniques and the knowing what’s possible, my goal is to help you get the most out of your investment in SharePoint. I think SharePoint has some really compelling capabilities, but I think it’s also kind of a shame that a lot of enterprises under-utilize that and that’s part of what we’re hoping to do in this webinar series is not only raise some awareness about ThreeWill, but also some awareness about some of the cool things that you can do with SharePoint inside your company.

With that, what are we going to talk about today? Well, SharePoint is all about collaboration. That’s one of the principle reasons that it exists. We’re going to talk about SharePoint in light of another tool that we use for collaboration and that is email. Whether it’s right or wrong and whatever the reasons for it, I would make the argument that this is the way that most of us collaborate in business. We put an attachment on an email and we ship it off and the process begins. What I’m going to try to do today is convince you that that’s not the right answer in all circumstances and that if you have access to SharePoint, you have a much better mechanism at your disposal for doing that kind of collaboration.

Then from there, we’re going to build on that and talk about what we call around here “a single source of truth”. If we’re going to collaborate on something, then we all kind of have to agree upon what the truth is, what’s the actual data, and what state that data is in. If you’re not already aware of it, I think you’re going to see that SharePoint gives us a really, really easy way at arriving at that. Easy enough, in fact, that we don’t even necessarily need to involve a developer for this part. If we can get our SharePoint users to the point where they understand and can implement this part of the story, then the next part gets pretty interesting and that is the producing of documents.

What I’m going to do is talk you through a couple of application scenarios where we can take data that’s been stored in SharePoint and then use that to produce PDFs or spreadsheets or whatever it is we’re trying to create, and in doing so, automate a repeatable business process. That’s kind of a cornerstone of what I’m talking to you about today. These are not things that are kind of one-offs, but they’re things that have to happen over and over and over. If we have to do a report on our quarterly sales numbers, we have to do that every quarter. If we can make that process simple and easy to consume, that’s just going to make life better for both those that produce the data and those that consume the data.

That’s really the over-arching story of this entire webinar. Even though we may talk about technical details or end-user details. Ultimately, it is about a repeatable process where we can automate the generation of documents that we use in the running of our businesses.

Then, with that, let’s talk a little bit about the collaborating via email model. I’m not trying to hide my disdain for it here, at least in certain circumstances. The reason for that, if we go back through the history of email, originally it was a text medium. It was just a way to send a message to someone and get a response. Really, I can remember in my career I first started using email, the thing that I thought was so awesome about it was that it was in asynchronous form of communication where I could send off a message and when this person got time, they could respond. I didn’t have to interrupt them by calling them on the phone, nor did I have to sit and wait on the phone while they tried to formulate an answer. In some respect, it’s a very, very efficient way for human beings to communicate with each other.

As email evolved, it got some extensions. Most of you are familiar with this acronym of multipurpose internet mail extensions or MIME. Now, several really good things came out of that. The ability to have different character sets. The ability to have some more sophisticated capabilities in mail. Depending on its intended use, one possible unfortunate side effect is the advent of attachments.

I know you guys love attachments. You use them a lot and I use them a lot, too, but I say “unfortunate” in a couple of respects. Really the problem with attachments is that they don’t require a lot of forethought. They don’t require us to put anything in place in order to carry out the process. We can just at the drop of a hat say, “Oh, you need that document? Boom, I’m going to send you an attachment.”

In a general sense, there are three, maybe more, kind of issues with this that I want to talk about. I’m going to focus on the first one. The versioning or the accuracy of the data. That is really hard to do when you’re working with attachments. Give Microsoft credit. Over the years they’ve tried to put some capabilities into their office products where they can track changes and be aware of the fact that it’s being circulated by email, but in the end, the problem here is just this wasn’t meant to be a collaborative authoring tool.

Even if you can get past that, another problem with attachments, and I have to confess I do this all the time, I know a coworker sent me something and I know it’s attached to an email, but gosh, which folder did I put it in or did I accidentally delete it? This whole accountability of being able to locate a piece of important data quickly. If I don’t religiously get those things out of the email and into some well-known place, it’s very, very easy for me to lose track of them. At that point, I’m either at the mercy of the organizational system in my inbox or I’m at the mercy of the search facility in my inbox in order to locate something that’s really important.

Another one that’s a challenge with email attachments, and I’ve experienced this firsthand, you’ve got someone that’s kind of responsible for some important data, they have it in their inbox in a series of attachments, and they depart the company. Before we really get a chance to contemplate what it might mean, their email gets decommissioned. Their inbox gets blown away. This data that might really be important to us is kind of lost there.

My encouragement to you in a very general sense is before you start attaching business documents and shipping them off, think a little bit. Think about maybe there’s a better mechanism that we could use to accomplish that kind of collaboration. Now, the flip side of that, if we’re not really going to be changing the data, if I’m just sending a map of directions to my friends so they can come to my party, attachments are awesome for that. I think it’s just another case of let’s make sure we’re using the right tool for the right job.

Then let’s take a typical scenario and I can say typical not only because of I’ve experienced this, but because literally we’ve written applications for several of our customers that follow this exact pattern. We start out with a series of sales managers. We’ll say that we have a manager of each one of our regions and each one of these managers has got to contribute to a spreadsheet of quarterly numbers. Ultimately, the consumer of this spreadsheet will be people that are high in the company, executives. It’s really important that the information be correct and up-to-date.

How we usually start this, well, one of the managers will step up and say, “Okay, I’ll start the spreadsheet.” He or she goes in and puts in their numbers and then they do an email and they send it to all of the other users at once. Then they all take a solemn oath that they’re going to make sure that they keep track of who’s making what changes so that we can keep everything up-to-date. The reality is that rarely happens. The original author forgot they made a change and they went back in and did it and now two other people don’t know about it and so in a more specific sense, we have multiple copies, multiple changes, but no single, authoritative copy.

It’s funny. I’ve seen people go to just great lengths to make this work. They’ll start attaching version numbers to the end of the documents or they’ll start attaching their initials and a version number. Again, that’s just all treating the symptom instead of treating the actual problem here. Then put yourself in the role of the VP of Sales. This person has now got to try to figure out which one of those spreadsheets has the most correct data and then they’ve got to use that and forward it to the executive team. If we’re willing to do a small amount of work up front, what we can do is make it so that this information can be more reliably entered, more reliably modified, and then more reliably reported to those that are interested in it.

Let’s look at the problem a little bit differently. We said that the goal here was to produce a spreadsheet that had quarterly sales numbers that were accurate and usable, but part of the problem is that we started with the end. Let’s don’t start with the spreadsheet. Instead, let’s start at a higher level. We have a scenario where we’re trying to author or create some content and we have many contributors to that. Whenever you think about problems in business and how you’re going to solve them with technology, that’s something you always want to consider. Who are my authors and how many of them am I going to have? In this case, if we have regions, we’re going to have five authors. If we have thirty regions, we’re going to have thirty authors. We need a mechanism that works as well for two as it does for twenty-two. Something that will scale in the human sense.

Another way of looking at this problem is not only do we have many data contributors, but we have a relatively-small number of reviewers or approvers. In fact, in this case we have exactly one. The Vice President of Sales. Not only do we need to have a mechanism that will allow many people to contribute, we also have a mechanism that will allow those who are responsible for doing so to be able to review, approve, and reject content without a lot of botheration.

Another thing that’s really important here, if we’re going to make this work, we’ve got to have one location for the data and there has to be a mechanism for managing the changes in that data. This is also a really important one. We want to do this in such a way that those who would consume the data can’t see it until it is approved. That way we don’t get illegitimate numbers out in the wild and then we have to play this game of, “Oh my goodness, who all has this and how do I chase them down and tell them how to fix the numbers?”

Only then, once we’ve gone through the process of authoring, approving, and making the data fit to be used, do we worry about the producing of the documents. Then what you’re going to see in the little demonstrations that I’m going to do for you is that in a very basic sense, we can use SharePoint itself to produce a document or we can write some applications that would give us a great deal of power and flexibility on how we produce those documents.

Then let’s contemplate for a moment how we’re going to use SharePoint to drive us toward this single source of truth so that we can all agree upon the data that we’re acting on. Well, in my scenario what we’re going to do is, instead of entering those numbers directly into a spreadsheet, instead we’re going to enter them into a SharePoint. As you’re going to see in the demonstration in just a moment, this is a really, really simple SharePoint list that any user can go in and create on their own. I think that’s one of the most cool things about SharePoint and it’s something that you really need to be tapped in if you’re an end-user and that is you can get this process rolling into a usable state without having to rely upon a developer. Then once you do get it to a usable state, we can bring the developers in and even automate it further.

The user’s going to create the list and then once the list is created, now we have an easy path for all of those that are submitting the data. Quite literally, we can circulate a URL and say, “From now on, when you need to submit quarterly numbers, this is the place you go.” Not only could we give them the URL of the SharePoint site, we can even give them the URL to the specific list. It really doesn’t get much easier than that in terms of them knowing where to go in order to submit the data.

As you’re going to see in the demonstration, SharePoint has some built-in approval capabilities that you may or may not be aware of. They really take two levels. At a very, very industrial-strength level we have this notion of publishing and publishing sites, but at the list and library level, we also have this concept of approval. It’s really a breeze to set up. It just says, “I’m going to make it so that list items are only visible to the public at large or to the consuming public once we’ve approved them.” By default, when the data is submitted, it’s in a state where it’s not going to be seen by anyone other than the contributor and those that are authorized to approve it.

Let me go in and just show you a little bit about how this is going to work. Now, a couple of disclaimers as I go through my demonstrations here. Disclaimer number one is that a lot of what I’m going to be showing you today, I’m actually running on a local VM on my system. Because I’m running it on a laptop, sometimes it might not be the fastest. I wouldn’t draw any negative conclusions about SharePoint on that. It’s just the limitations of running in a VM.

Another thing that I want to point out is as I go through my various demos, I’m actually running some of them against my local SharePoint instance and some of them against Office 365. I think the important point there is just whatever you’re going to learn today, it would actually apply equally to both of those environments. Whether you’re cloud-based or not, it’s not really going to matter.

Maybe a third quick point to make about my demos. These are all very-much focused on SharePoint 2013, however, many of the concepts and much of the code that I’m going to demonstrate could also be applied to a 2010 environment. Some of the application bits that I’m going to do would not be quite as cool or quite as integrated in a SharePoint 2010 environment, but the basic concepts would still exist.

I’m going to start with a very simple demo where I go into the contents for my site and I add a custom list. I’ll start the process by doing Add an App and then choose Custom List as my template. Then, for those of you who haven’t done this before, really all I have to do is give it a name and start customizing it. I’ll just call this, “Sales Numbers List,” and create it. Once I’ve created that list, I can navigate to it. I can click on the List tab and then go to List Settings. Now, obviously the presumption here is that as an end-user, you’re authorized to do this. If you’re not, then that’s something that you wouldn’t want to talk to your administrator about. You don’t have to be a site owner or an administrator to be able to change the capabilities of a list.

There’s two things that I would potentially do here in order to implement this. The first thing I would do is go down and create some columns. Obviously, I would want to go in, as example, and create a column for my gross sales with currency and my net sales and the year and the quarter and all those other things that I want to be able to track. Then once I’ve created the necessary columns, the other thing that I’ve got to do is set up my approval mechanism. Again, this just couldn’t be easier because literally I’m going to go into the Versioning Settings and right here, I’m going to turn on Require Content Approval for Submitted Items. That’s literally all that’s involved in getting this thing up and going.

Now, kind of like they do on the cooking shows here, I’ve shown you the basics of how I would do this. Let me know go over to a site where I’ve actually got it set up and I can show you this list in action. This particular one is named, “Quarterly Numbers,” and you can see that all of my sales managers have done a really, really good job of getting in their numbers for the third quarter. Now we find ourselves approaching the end of the fourth quarter and I’m going to put myself in the role of a sales manager and I’m going to go in and add some data. I’ll just simply go up and add a new list item and then notice the warning that the user gets here. It basically says, “This is a list that’s using the approval mechanism, so when you submit your data, it’s only going to be seen by you and those that are eligible to do approvals.” I’ll talk about that a little more in just a moment, those that are eligible to do approvals.

I’ll go ahead and put in my fourth quarter numbers here and we’ll choose a particular number value for each one of these. Again, I don’t pretend these are accurate numbers. I’m just trying to put in something for demonstration purposes. Now, notice if you will, when I save this, it immediately goes into a pending state. I can see it as an author and then anyone else that has approval permissions would be able to see it.

Now, if I switch roles real quickly, now I’m in the VP of Sales role and I’ve got to go in and make a decision about whether this thing is legit or not. I’ve done a little research and what I’ve discovered is that this number is actually incorrect based on some other things that I’ve seen. I’ll select the item, I’ll go to the Items tab on the ribbon, and then I’ll choose “Approve and Reject”. What I can do is set it to a rejected state and then put in some details about what’s wrong with this and what needs to be fixed. It’s a very simple process. Those that are contributors can easily see whether their data has been approved or not. In fact, they can even set up an alert on this list to let them know when data has changed that they care about. Later on, if they do realize their mistake and they come in and fix it, it’s a pretty simple matter. All they’ve got to do is go in and edit the item and recognize that they put one too many zeros on that and save it. Now it’s gone back to a pending state and someone could go in and approve it.

Those of you out there that are more on the end-user scale of things, that’s a ton of functionality and, again, it’s maybe ten minutes’ worth of work to set it up. If you’re not already familiar with those capabilities of using a list that way, I want to strongly encourage you to dig a little deeper on that and give it a try. I think what you’ll see is that you can go a long way toward accomplishing what it is that you’re trying to accomplish.

Now, the last thing I wanted to show you is how we get permissions for these. I know that some of you would be the ones that would set up permissions. Others of you, you’re not going to be doing this, but you just need to know a little about it so that you can ask the right questions. If I go to my Site Settings page and then I select Site Permissions, what I want to show you is just real simply how you might set this up. What I can do is I can set up a group for people that should be allowed to approve the content. I can name that whatever I wanted to name and I could put in whatever individuals I wanted to put in.

The key part, though, is right down here. Notice the various permission levels. Obviously, we don’t want to give them full control, but in order for someone to be approver, they have to have at least design-level permissions. Obviously, my VP of Sales, he or she is going to be in the role of approving this content. They would need to have this effective permission in order to do that. Most likely, those that are contributing data are just going to have contribute capabilities. Those that might be responsible for configuring lists would need edit capabilities. All of it’s there and it’s out of the box, you just got to know how to stack things up correctly in order to take advantage of it.

Now that we’ve reached a point where we can pretty easily capture that data and feel pretty confident that it’s going to be accurately disseminated, let’s now talk a little bit about how we can get at the data. If you’re lucky enough to have a developer at your disposal, we’re going to talk in a few minutes about some of the amazing things that you could do with that. Even if you don’t have a developer at hand, once you get the list looking the way that you want it to look, it’s a pretty simple thing to get it into Excel. Literally, I’m going to go over into Quarterly Numbers and I’m going to go to the List tab, and then I’m going to choose Export to Excel.

The cool thing about this approach is that it’s not actually downloading the data, rather it’s downloading a query that pulls that data from SharePoint. The idea here is I can now slice and dice this data and accomplish whatever I want. What I would tell you is that in the real world, this is an approach that’s really cool and probably okay for someone that’s a bit of a SharePoint jockey, but for someone that’s more maybe in an executive role and they just want to push a button and have the stuff at their disposal, this probably draws them a little farther into the SharePoint experience than they want to go. For that audience, that’s the case at which we’d probably need to think about writing some applications. Let’s go have a look at that.

How is it that we’re going to produce our documents? Well, in a very general sense, what we’re going to do is write an application that uses SharePoint as its source of data. No doubt many of you have either written or worked on applications where someone wrote a “front end” that knows how to read out a SQL server or some line of business application. We’re essentially doing the same thing. In this case, instead of reading the data and putting it on screen in a data entry form, these are apps that are going to read the SharePoint data and produce some kind of document from it.

I’m going to show you two approaches that you can use to accomplish this in SharePoint 2013. The first of those is a SharePoint 2013 app. The advantage to this one is really kind of multi-fold. Number one, it integrates directly with the SharePoint site. The same destination that someone might go to to enter data, they can go to that same destination and with a couple of mouse clicks extract the data that they need.

Another thing that’s nice about the SharePoint 2013 app approach is that because it has some knowledge of SharePoint, not only do we have the ability to programmatically read data out of the SharePoint list to produce the document, but then we can turn around and take the produced document and store it in the document library. The person responsible can generate that document, but then any other party that’s interested in it can know that if they go to that library, they’re going to see an approved and released version of that document.

The other approach, and this one is completely new as well, something called an Office 2013 app. I have to confess, even though these have been out for a little while, it’s only now that I’ve gotten an opportunity to start working with these. I have to admit I’m actually pretty excited about some of the things that you can do here. The idea with this is instead of having some kind of app that integrates with the SharePoint site directly, instead I’m going to write an app that actually integrates with the desktop version of Excel or PowerPoint or whatever app I’m writing it for. The really cool part of this, as you’ll see in my demonstration, is that the consumer doesn’t have to visit SharePoint. They don’t have any concept that SharePoint is even coming to the party. All they know is that they can launch their Office app and make a few mouse clicks and wind up with the data that they want. I think you’re going to like what we can do with those.

I know some of you have been to prior webinars or maybe you just read up on your own enough that you already know about this, but on the chance that a few of you kind of don’t know the SharePoint 2013 app story, let me just take one slide here and talk about that briefly. First of all, when we say “app,” we kind of joke around the office of App with a capital “A” versus app with a lower case “a”. This is an app with a capital “A”. It’s a particular kind of programming model that’s new for SharePoint 2013. Apps are created by developers. This is not a power user thing, but rather a developer thing that is created on behalf of users. When we write SharePoint 2013 apps, we’re writing code that runs out-of-process or, perhaps betters said, runs outside of SharePoint. There’s a lot of reasons why that’s a really good thing, not the least of which is it kind of insulates our SharePoint server from being slowed down or brought to its knees by bad code.

From a developer perspective, when I say “out-of-process,” what does that mean? Well, it really means your code is going to be running in one of a couple of places. It’s either going to be JavaScript that’s running in the browser or it may be code that’s running on another server. Depending on the kind of server it is, it might likely be .NET codes such as C#. It’s also possible that you could have a non-Microsoft server that’s integrated into this process. Either way, you could just think about this model as being, “I’m going to write my code and then that code is going to remotely call into SharePoint and get whatever data it needs.”

How do you users get these apps? There’s really two models. First of all, you would have something within your enterprise called the App Catalog. Essentially, the App Catalog is like a giant library where all authorized users can find apps and install them to their sites. There’s also a global SharePoint Store. Most companies probably wouldn’t open up to their users for fear of what they might download, but the cool thing about the store is that as a user, I can actually visit there, find an app that I like, and then make a request for it in my company’s app catalog. At that point, an administrator would see that request, see that I want that app, they could go review it, and if it makes sense, they could actually purchase it and put it in the app catalog.

There’s a lot of exciting things going on there. For users, your developers are going to be able to build you some really cool things that are very easy for you to consume. For developers, what you just got is a much better mechanism on rolling out and updating your apps and having a central place where people can actually discover those apps.

For the specifics of the app that I’ve written for you that I’m going to demonstrate here in just a moment, we’ll start with the list that I showed you just a moment ago. I’ve got a SharePoint list and it contains sales numbers for a variety of quarters and years. Then from there, I’m going to write what we would call a cloud-hosted app. That is to say, I’m going to have some server code that runs in another server. It’s going to call in the SharePoint and then it’s going to read the sales numbers from that list for a particular year and for a particular quarter. Then those of you that are developers, what I’m using here is the C# client-side object model. I’m just making a remote call into SharePoint.

Now once I’ve pulled down the data that I need from SharePoint, the next step is to produce the PDF in this case. I’m going to do that via a third-party PDF library. Once I’ve actually produced that document, I’m going to turn right around and have my application write it to a document library and in that same SharePoint site. Then from there, the users will have two approaches to getting at that data. For the user that runs my application, they’ll actually be presented with a hyperlink that will easily let them read that document. In addition to that, anybody on the team that has access to that particular document library can navigate directly to it and see all of the contents within.

All right. Let me take a moment now and show you how this works. Now, those of you that are developers, you’re probably very excited to see that I’m in Visual Studio now. Those of you who aren’t developers, don’t let your eyes glaze over. I’m not going to stay in here very long. I’m just using this as a point to kind of help our developer friends understand what I’m doing here.

For the sake of our demonstration, I want to just focus on this page right here, default.aspx. This is going to run in my remote web and it has a very simple UI that’s just going to allow the user to pick a year, pick a quarter, and then generate the information that they’re interested in. Now, if we go and look at the code behind that is associated with this, even if you haven’t done any SharePoint 2013 programming just yet, if you’ve done some client-side object model in SharePoint 2010, it’s essentially the same here. The ultimate goal is for me to create a client context object. This is what will allow me to call into SharePoint and request the data that I’m interested in.

In this case, the data that I’m interested in, I’m going to extract that via a CAML query. Even if you have no idea what a CAML query is, basically we’re saying, “I want a particular year and I want a particular quarter.” Those are going to come from whatever the user happened to choose on the screen. Once I’ve asked for my data and received it back, then I’m just going to write a simple loop and in that loop I’m going to take that data and render it into a PDF. I’m going to take that PDF and write it to a memory stream. Then finally, I’m going to upload that file. Literally, I’m going to use the client-side object model to take a document that’s buffered in memory and upload it.

There should be two reactions to that. If you’re a developer, that’s pretty cool. I like the fact that I can do that. Then if you’re an end-user and you’re a little bewildered, let me kind of set your mind at ease. The reason you care about this as an end-user is that you didn’t have to think about where you stored the app. You didn’t have to worry about if you stored it in the right place or the right location on your hard drive. The app was responsible for putting it in the correct place and your job was just to ask for what you want and then get it. It doesn’t get much better than that.

All right, with that, let me go ahead and run this. Again, obviously those of you in the user role, you would not be running it from Visual Studio. This app would already be installed on your SharePoint site and you’d be able to navigate to it. Now, if I attempt to generate data for which I don’t really have any data, it’s just going to give me a little message saying that we don’t have any numbers for that particular time unit. Instead, let’s go get the third quarter of 2013 and I’ll click on “Generate”. Notice our users get two things here. They get a little indication that the PDF has been generated and then they also have this click-to-open link. As I click on that, you can see that I literally generated a PDF that just gave them a formatted list of the available numbers.

Really important here. Don’t get hung up on the format of the document. I wasn’t trying to make an artistic statement here so much as I was just trying to show you that you can programmatically create these things and upload them to SharePoint fairly easily. Obviously, we have a particular look and feel that we wanted to achieve and we can certainly do that on behalf of our users.

Now the other thing to think about is if I returned to the site on which this app is running, at any future point, if I want to see this document, I can literally go into the document’s library and there’s the PDF that I just generated. Couple of things that I think are kind of cool about this, not only did I generate the document and it knows that it was me that generated it, there’s another little cool part here and that is, notice that last column, “App Modified By”. Those of you that are developers, you probably want to make a mental or written note of this. When you start writing apps, SharePoint not only has the concept of the user’s identity, but also the app’s identity. There are some really, really clever things that you can do with that. That’s a capability that I think is pretty exciting and holds out some interesting ways to do things on behalf of the user.

All right, so there’s the process. Going forward, how would our users consume this? They’d consume it in one of two ways. They would either go to their SharePoint site, go to the Site Contents area, and click on the app in question, or we could even simply provide them with a link. They use the link and it takes them directly to the app.

I’m going to keep plugging ahead here. Just an FYI, I noticed maybe one or two of you have posted some questions out there. That’s awesome. Please continue to do that. I’m going to save those until the very end just so that I make sure that I get through all of my material. Please don’t think I’m ignoring you. I’m just holding those until the end.

All right, so the SharePoint app is cool enough, but let’s face it. There are some people out there, right or wrong, that are just a little biased against SharePoint. Maybe they had a bad experience. Maybe they just don’t like the way it looks. Doesn’t matter. The point is we can implement this same model that we just saw, a very convenient way for a consumer to get at the data that they want without them ever having to visit SharePoint.

That’s really what the Office 2013 apps model is going to be about here. Conceptually, it’s similar to SharePoint. You’re writing an app. It’s going to enhance this product or customize this product that Microsoft is providing to you. The difference here is instead of integrating through the web browser onto a SharePoint site, it’s actually going to integrate directly with Word or Excel or PowerPoint or whatever you target this to.

When you write these apps, what you’re really doing is you’re writing a web page. Developers, everything you already know about server code and HTML and JavaScript, those would all still apply. The difference here is that the web page that you’re authoring is actually going to be visible or hosted inside of Excel or inside of Word. As you’ll see in the code that I’m going to demonstrate for you in just a moment, you’ll actually have a JavaScript library that will allow your page to interact with the hosting application. I can put together logic that would allow me to inject data into a spreadsheet. I could put together logic that would let me read data out of a Word document and then do some kind of action based on that.

What is the integration point here? Well, because we’re using standard-based web tools here, we can take these pages that we’re going to add to Word or Excel and we can integrate them with a variety of back-end data sources, most notably for this webinar, SharePoint. The idea here is that the user’s going to be completely sheltered from SharePoint. They’re not even going to know it’s a part of the party, but we’ll be able to go out to SharePoint on their behalf, pull down whatever data they need, and present it in an easy-to-use format.

Now there are essentially three different kinds of applications that you can write for Office 2013. The first of those is called a task pane app. That’s the kind of app that I’m going to be building for you. As you can see, the app renders in a Window to the right of your main document and then, as I mentioned a moment ago, this app is going to have both the capabilities of putting things into the document as well as reading things out of the document and taking some kind of answer on them. Then this back-end data source, you can see in this little simulated example, someone’s actually written a task pane app that will take text, call Bing, and translate it. Pretty much any back-end service that’s a breachable service by a REST call or a web service call, we’re going to be able to do those kinds of integrations from within Office.

A second approach is one where we right what’s called a content app. In this case, instead of being in a separate pane, this is actually an application that embeds directly in the document itself. In this example, we’ve actually embedded this app that someone has authored that will read data about states and sales and then actually give us a graphical map that shows where those sales are taking place. Maybe another way for you to think about these if you’re a developer, anytime you’ve got a user that’s trying to write a formula that’s really, really complicated in Excel or they’re trying to create a chart that has some highly-customized capabilities, it could be that you could author a content app on their behalf that would do all of those things and for them, it’s literally a couple of clicks to put it in place and use it.

Then, ironically enough, I kind of ragged on mail and attachments earlier in the presentation, but there are some cases where we’d want to be able to do some clever things with email integration. That’s what a mail app is about. A classic example here. As a user, I could highlight addresses and then I could have a mail-out that would actually go out and do a Bing look-up and show the user the map for that. Anything that you’re doing that is legitimately built around the email process, we can do some very clever things with the mail app.

Once again, for those of you that do better with pictures as opposed to words, we’re starting in the same place here. We’re starting with a SharePoint list that contains the sales numbers that you saw at the beginning of my presentation. Now in this case, instead of the user navigating to a particular place, they’re going to launch Excel. Then their Excel instance will be configured in such a way that they actually have access to this task pane app that I’ve written on their behalf. For them, it’s as simple as picking the right options, clicking a button, the task pane app then will go out and read the SharePoint data on their behalf, pull it out, and then populate it into a spreadsheet or whatever kind of office document that we’d like to put it in.

Let me give you a couple of demonstrations here. I’m going to show you how you can use this process to generate an Excel spreadsheet. I’m also going to show you how you can generate PowerPoint data. I can tell you again, literally, we have written apps on behalf of customers that do both of these things. The story here is just now that we’ve got these Office 2013 apps, the process gets really, really simple and really, really easy.

All right, so I’m going to go back in now and I’m going to open up the first of my two demo apps. Then, for the people in the room that are more on the developer track, you’ll notice that when you create an office app, structurally it looks very similar to a cloud-hosted SharePoint app. That’s a pretty decent analogy. To put that in another way, most of the code that you’re going to be writing is going to be down in this app web and then really the top part is just declaratively describing some information about your app. Among the things that it would tell you, what products can it work with. The fact that this says “workbook” here, that tells me that this is for Excel and that it’s not for PowerPoint. They don’t have to be mutually exclusive. You could actually write an app that would integrate with Word and Excel and PowerPoint all at the same time.

Then the other part of this, the actual logic here for creating the data and pushing it into the document. Again, I don’t want to belabor this too much because I know I have a mixed audience here. Those of you that looked at the code earlier, here’s what you need to take away. I’m using the exact same client-side object model code to fetch the data. Same query and everything. The only difference here is that once I’ve got the data, I’m going to format it in adjacent array format and then send that back as a string. Then I’ll actually have some JavaScript code that can take that, decompose it into an array, and then insert it into the document.

How is it that things get into the document? For those of you that are going to be writing this kind of code, this is probably the most important sentence of all in this demo and that is everything that allows my app to interact with Office is encapsulated in this office.js script. Literally it’s providing me with an object model that will allow me to read and write data from JavaScript down into the document or vice versa.

In this example, I’ve set my data back in a container named “Payload”. I’m going to take that, parse it into an array, and then notice what I’m doing in this example. Because it’s Excel, Excel has this concept of a matrix. To put that in simple terms, you can create a two-dimensional array, put data in it, hand that over to Excel, and Excel knows automatically how to format that data correctly. The method that I’m using here is called “Set Selected Data ASYNC”. Those of you that write a lot of code, you know that when you see that word “ASYNC,” you can optionally provide a callback that would let you do a success or a failure message. For this example, I’m leaving off the callback and I’m just putting the data in directly.

All right. Let’s give this a try here. I’m going to run it now. Again, those of you that are in the user role, you’re not going to run this from Visual Studio. You would just have Excel already configured to have availability to this app. Notice what I get here. When I run it, I’m running Excel. Then the markup you saw just a moment ago has now actually rendered in this window. If I try to get data for a year-end quarter that doesn’t have any associated data, I get the same message that I got before. On the other hand, if I go in and pick some valid settings here, notice that it retrieves the data and immediately puts it into the spreadsheet.

Now if I’m in the role here of the Sales Vice President, watch what I’m going to do. I’m going to go right here and select some data and I’m going to insert a recommended chart and maybe change a title or two and I’m done. Save my work and publish this information to whoever needs to get it. Then when I’ve got to do this again next quarter, I fire up Excel, I create a new spreadsheet, I click the mouse the same three times like I did before, and I’m done. To me, the exciting part of this is just that we’re getting the collaborative capabilities of SharePoint to manage the data, but for those who want to consume it, we consume it very close to the source. They don’t have to know that much about where it came from and how it got constructed.

The model for how we would do this on PowerPoint is almost identical. The one little bit of code that’s going to be different here is that PowerPoint really doesn’t understand the concept of anything other than text data. You’ll remember a moment ago when I wrote my data, I had a coercion type of matrix. In this case, I’m just writing string data. Literally what I’m doing here is just writing a loop that will construct the data and then I’m going to inject that into a PowerPoint document.

Let’s suppose that you’re the VP of Sales and you have to produce a PowerPoint every quarter where you report on how your divisions are doing and then one of the slides has to have your numbers. Well, you’re in PowerPoint. You’re authoring the different parts of the PowerPoint and then once you get to the point in time where you want to add that particular slide, you just simply come over here and click “New Slide,” and then you click in the body of the slide. You choose the year that you’re interested in and the quarter and you click “Get Data”. Notice what it did. It actually read that data from SharePoint, formatted it in a way that is appropriate for my PowerPoint, and so now all I have to do is put on a title and then continue the process. Again, I think that is a really compelling story on how you can generate these things very easily.

All right, let’s hit a few wrap-up points here. We’re getting near the top of the hour. Kind of recapping what we talked about. First of all, if you know you’re going to be collaborating on something where there’s going to be more than one author, don’t use email. Spin the ten or fifteen minutes that it would take to set up a SharePoint list or a library and keep the data there. You’re just going to be way ahead of the curve even if you don’t have the luxury of writing applications. That single source of truth. Really in this one, I focused on lists because the data was more appropriate for a list. If you truly have to author a spreadsheet and that’s the only way you know to do it is just to author the spreadsheet, you still ought to be keeping it in a library as well. Then you can use the check-in and check-out mechanism just to make sure that you’re not trampling each other’s changes. These are going to be much better alternatives than trying to track it by email.

Another thing, take advantage of that SharePoint approval mechanism. Literally one click and you’ve got yourself in a circumstance where people can’t see the data until it’s been legitimized by some reviewing party. Then from there, for those of you that are developers, I hope I kind of whetted your appetite a little bit there in terms of writing SharePoint apps and Office apps as a way to generate these documents. Then, if you’re more on the user side of the house, while you’re not going to be authoring these applications, you now kind of know what’s possible and this is something that you can take back to your developers and kind of blaze a trail on getting these things done. Obviously, part of the point of these webinars is just the fact that ThreeWill does this. We’ve done this for several our customers and we’ve got a really good track record in making it so that people can get these documents produced repeatably and reliably without having to exert a lot of effort.

I hope enough of you stayed awake to this part of the presentation to kind of hear a little bit about our giveaway. Again, in order for you to be eligible for the giveaway, not only did you have to attend the webinar, but you’ve got to properly complete the end-of-webinar survey. You’ll get prompted for that when you exit the webinar. If for some reason you miss that or you lose that link, you can email me and I can send you an alternate link. I encourage you to go ahead and do it. Notice it says, “properly complete.” There is actually a question in there where you have to say, yes, you’d like to be registered for the drawing. The reason that we’re doing that is that we occasionally have customers that work for companies where they’re not allowed to receive gifts over a certain amount. If you’re at a company where you’re not allowed to do that, you may want to forego the contest, but for the rest of you, that’s how you’re going to opt-in.

A couple of details. We’ll perform the drawing tomorrow, so that will give you the rest of today and the evening to get the survey filled out. It shouldn’t take long at all. It’s a very brief survey. We’ll announce it in a couple of ways. We’ll announce it on our Twitter feed. That’s @threewill. We’ll also announce it on our Facebook page, Then just as a follow-up, we’ll also notify the winner via email. Then pretty much like any contest, the decision of ThreeWill concerning the giveaway is final.

With that, I’m at the end of my prepared material. For some of you, you’ve got a busy day ahead of you and you’re going to launch onto other things and that’s great. For those of you that can stick around a second, I’m going to see if I’ve got any questions. You can post those questions right now if you haven’t posted them already. If I have any, I’ll read those out loud.

I didn’t get a question so much as a comment. A couple of you thought that the Office 2013 app integration was pretty cool. I completely agree. I think that there are some enormous opportunities there for you to integrate data and make it very easy for your users to consume and let them be largely naïve about where that data is coming from. They just get what they need and they don’t have to do a lot of clicks.

Again, I’m going to stay on the line here for a couple of more minutes. If you’ve got your questions, please add those to the question manager and I’ll respond to those out loud. Otherwise, I really appreciate your time. I know that any time you invest an hour of your work life, you’re hoping to get something out of it. I hope that you’ve seen some things here that would apply to your organization. If you’ve got any questions or any interest whatsoever in letting us help you out with that, feel free to drop me an email. Again, [email protected] and I’ll be sure and connect you to the right place. Thanks again for attending.

read more
John UnderwoodSharePoint Document Generation Webinar

Enterprise SharePoint Search Webinar

John Underwood is a Technical Evangelist at ThreeWill. He has nearly thirty years of software development experience. He is an experienced technical instructor with superior presentation skills and is proficient in delivering standard curriculum as well as developing and delivering custom curriculum.

Hello folks, John Underwood here welcoming you to another of the ThreeWill Webinar series. Today, we’re going to talk about Enterprise SharePoint Search. I hope you’re having a good day wherever you’re at. I’m coming to you from just North of Atlanta, Georgia in Alpharetta and we’ve actually had a rare sunny day around here. It’s been a pretty good day. A little bit about me. For those of you who’ve not done events with me before, I’m lucky enough to be an employee of ThreeWill. My job title here is technical evangelist. That means I get to do events like this. I also do training and I even get to work on the odd project every now and then. A couple of ways to contact me obviously, you can use my email address on the screen. If you are someone that enjoys being a part of Twitter, there’s my Twitter handle. You’ll get a nice mix of hobbies and information, but mostly, you’ll get some good SharePoint information. I hope you’ll take advantage of that and follow me if you’re inclined to do so.

I know we want to get right into the material, but a couple of quick housekeeping items before we start. First of all, today’s session is being recorded. If you find that you have to drop off in the middle or if you’ve got some co-workers that were unable to attend but maybe want to see the information later, it will be available online for a certain amount of time. Essentially what you do is go back to the same URL that you used to originally visit and register for the event. You can take that invitation that you received and you can forward that to any of your friends or associates and they’ll be able to use that in order to play the video. Again, it’s interesting, even if they didn’t attend, they’ll still be able to register and view the video.

Then, one other thing. Although we hope we won’t have any technical interruptions, it’s not impossible that a phone would get disconnected or the power might go out temporarily or something like that. If any of those kinds of things happen during the event, you can notify me that you’re no longer seeing me. If you just send an email to the address you see there on the screen, that comes directly to my phone and at least, I’ll know something’s going on and we can see if we can’t get service restored and resumed. Again, that’s rare that that happens.

All right, with introductions and housekeeping items out of the way, let’s talk about what I’m going to cover with you today. First of all, I’m going to talk a little bit about search-driven applications. I think you’ll see that if you don’t already know about these, they’re going to represent what is for a lot of companies a missed opportunity. The ability to really exploit the SharePoint search engine in a way that’s going to be beneficial to you. Then, after that, we’re going to talk a a little bit about extending and customizing search.

A few of the things we’ll see here is something where things that maybe a power user could do but a lot of these are going to be more developer or even administrator oriented. Really, what I hope to do with this part of the presentation is twofold. Number one, just letting you know what’s possible and number two, take an opportunity to show you some of the things that ThreeWill has done as far as extending and customizing search on behalf of our customers.

Then, finally, I’m going to wrap up with just a quick look at some of the new things in SharePoint 2013 as relates to search. I know inevitably when there’s a new release, there are a few companies that jump on it immediately, but there are others that take their time. Since I’ve been working in the SharePoint universe, I know that’s particularly true of SharePoint moving up to a new version is not trivial.

Really, more than anything, I just want you to know what’s coming and really plan for some of those opportunities at the risk of sounding like a Microsoft product personnel, will tell you that some of the new search things in SharePoint 2013, I think are really compelling. They alone may not be a reason enough for you to upgrade, but taking in a whole lot, I think it is a worthwhile upgrade that you’ll want to undertake as soon as possible.

Then, time-permitting, we’ll do a little Q & A at the end. If time doesn’t permit or if you don’t think of any questions at that point, of course, I always welcome questions in the future at my email address, so please take advantage of that.

All right, so let’s spend the first part of our time together now talking just a little bit about search-driven applications. The hope here is that if you haven’t already thought about the search engine in this way before, after we’re done here, you’re going to see it as more than just a place where you type in words. Hopefully, you’re going to see it as a repository of some really valuable information for your company.

Let’s start with a common scenario. An application that we might need within our company. Suppose that we work at a company. We use SharePoint pretty heavily. We’ve got a corporate home page that’s hosted on SharePoint and then we have lots of site collections and sites and sub-sites and teen sites and all these different things. We have a legitimate need to present a list of sites that a particular user can see. Again, it could very well be that a user has access to a useful site and they don’t even know it. They don’t even know what’s out there.

One of the ways we might solve this problem is by writing a Web Part. Again, I’m a developer and what do developers do? They develop. They write code. Almost inevitably, if someone brings a business problem to us and says, “How are you going to solve this?” The first little thought balloon that pops up over my head is, “I’m going to write some code.” In this case, we’ve got a developer that creates a Web Part and that Web Part iterates through sites and makes a list of available sites based on the current user.

Now, we all know and love this phrase, “It passes the developers works on my machine test.” What we see here is that, a user or excuse me, a developer that’s written this may be testing it in an environment where they have a dozen sites. Then from there, we deploy it to our company’s home page where we have thousands of sites. All of a sudden, somebody goes to the home page and it’s taking forever to load because this little application is now going to try to iterate through those thousands of sites for every single page visit to the home page.

There’s really a couple of things wrong with this. Thing number one, obviously, is that, this is going to be a huge performance hit and people are going to hate SharePoint when it’s really not even SharePoint’s fault. The other thing here that’s a little more subtle but perhaps even more important and that is, there was a huge missed opportunity here. Because of the preconceived notion of how this developer viewed this problem, they didn’t see that there’s actually an efficient, simple and easy way to solve it. Easy enough in fact that in some cases, we could even rely upon a power user to do that.

What I’m talking about there is really a search-driven application. What is a search-driven application? Well, in simple terms, instead of us trying to use the lists and libraries in our SharePoint world as a source of data, we’re actually using the search engine or more correctly, the search engine’s index as our source of data. Because after all, that index contains a lot of really useful things.

Now, when we talk about a search-driven application, we’re talking about a scenario where we’re not going to go ask the users to type in some data and search for things. Rather, we’re going to search on their behalf. Really, they may not even know that it’s coming from search, and they may not even see the query. We may do it in a very subtle way so that that information is hidden from them. All they know is that they are able to undertake some actions such as clicking a link or visiting and web page and they get whatever it is that they needed.

Another nice thing about search-driven applications is that, the results are security-trimmed, by the very nature when you go into SharePoint and you search for things, you only get results for the things that you are authorized to see. In the case of as an example trying to show a user all of the sites that they have access to, by definition the search engine’s not going to let us serve any site references up to users unless they’re authorized to visit that. That just means there’s a whole level of coding that we often have to undertake that we’re just going to be able to ignore here because the search engine and its index already knows the answer to that question.

Then, another thing about a search-driven application, and while this isn’t a requirement, I think it makes it more compelling and it’s really nice. That is, results are presented to the user in a way that are ready to work with. To say that a little bit differently, certainly we could take users to a search page and we could teach them how to look for the thing that they want to look for. With the little effort on our part, we can customize the output from that process in such a way that they don’t get a lot of the other decoration that we would normally see on a search results page. Instead, they’re just going to get the data in question and the importance of that data to them.

When I do some demonstrations on this in a few minutes, I’ll reinforce what I’m talking about when I say presenting the results in a way that are ready to work with. If you’re a developer and you’re on this webinar today, here is my encouragement to you. When you start thinking about the sources of data that are available to you when you are customizing SharePoint, don’t forget the search index. Even furthermore, almost any time you’re writing some kind of customization that’s asking a question, the odds are pretty good that the search engine would know the answers to those questions.

Let’s dig a little deeper then. If it’s all about asking the right question, then how can we ask the right question? At least traditionally, when we think of a search engine, we think about a keyword-driven search. As an example, excuse me, a user might type in the word invoice hoping to find things that are related to invoices or resumes, or a series of what we would call keywords. However, the thing that a lot of people don’t know and that is really useful for the search-driven application, and that is SharePoint has a capability that allows us to specify the kind of thing that we’re looking for. Not only might we say, “I want invoice,” but we might say, “I want invoice and I only want that word where it appears in a list or where it appears in a library.” We can refine the kind of thing that we want to get back from the search engine.

If you think a moment ago about my example of trying to give a list of sites that a user has access to, we could specifically say to the SharePoint search engine, “Hey, I only want you to send me back things that represent sites. Anything else, just leave out of the results.” Then, another capability that we have in the search engine that not everyone knows about, we can also qualify details about the item that we were looking for. Not only could I say, “I only want items that appear in a task list.” I could go further and say, “I only want the ones that are authored by Kim or the ones that are assigned to Kim.” This notion that the entities that we’re looking for out there, each have some properties associated with them, and the query language actually gives us some mechanism for specifying this property.

Then, let’s be a little more specific. How is it that we’re going to qualify the type of results? Well, one very easy one and again I’ll be demonstrating this in just a moment, is the use of IsDocument. That’s pretty straightforward. It just lets me say, “I either want to explicitly return results that are documents, or are not documents.” By doing that, I may be able to go from a search that gave me a whole lot of results, to a search that gave me a set of results that are a little closer to what I’m looking for.

Now, the next one, and in my opinion, simultaneously both the most important one and the one that’s probably the most underused is the concept of a content class. You may not be aware of the fact that when you issue a keyword search in SharePoint, you’re actually able to specify the content class that you wish to receive back. Even if you’re not a real SharePoint expert, you can figure it out based on what you see here. As an example, I could say, “I want to look for the word invoice but I only want to see back a list item.” Or, “I only want to see back something from a document library.” Then, we can even go a step further. Not only could we say, “I only want to see items from a list,” but we can even provide a type that says, “I only want to see items from a list that’s a task list or a contacts list, or even a calendar.”

When you begin to think about this ability to not only ask for what you want but to specify the kinds of things that you want, that’s really a cornerstone of building these kinds of applications. Then, I handed that just a moment ago, this is just a more formal look at that notion of a property restriction. In effect, we’re saying, once we found the things we think you’re looking for, we can even be more specific about, it should have a certain file extension. It should have a certain file name. It should have a certain author and so on and so on. Taking what we already know about keywords and laying on these layers of qualification, that’s how we’re going to be able to not only write better queries to start with, but then take these queries and package them up in a way where users can conveniently consume them.

Let me take a moment now and do a quick demonstration for you. I’ll show you a little bit about some of these restrictions. Now, just a little FYI here. I’m going to be working in a virtual machine environment and I’ll just confess to you right upfront. It’s running on my local machine and it’s not got the most horsepower in the world behind it. Don’t try to draw too many conclusions here about the performance of this application. Really, let’s just focus more on what it does. Now, those of you that have worked a lot with SharePoint, probably recognize the fact that I’ve got a SharePoint 2010 Team Site that I’m working with here. You can see I’ve got a top-level site named Acme Corporate. I’ve got a couple of sub-sites underneath. Then, I’ve got a search center setup as well.

Let me see if I can demonstrate now some of the techniques that I’ve talked about in terms of refining your search and then from there, we’ll put that all together into a simple little application. Let’s suppose that I’m looking for a resume and I know that it was authored by Kim. I can start with a real simple search just by putting in the word resume and then searching for that. Then, we can see that, I get back results here that are all over the board. I’ve got different things that are actually resumes. Not only that, I’ve even gotten pages that just happened to have the words resume on them.

Now, if I go a step further, and I can say, “Well, let’s refine it so that it looks for both of those words.” Certainly, that gets us a little closer but you can see that I’m also winding up with a little more debris down here that’s not really relevant to what I’m looking for. Again, in this somewhat contrived example, I’ve gotten it down to three items. I could probably live with that, but those of you that work in very large corporate environments, the little refinement we just did there maybe took you from 10 thousand results to three thousand results. That’s a huge improvement, but you’ve still got a big hole to dig out of.

Let’s refine it just a little bit more. We want to look for the keyword resume, but we only want to get back items where the author of that document is Kim. You can see now that we’ve got a much better result at this point. We’ve got the actual resume and we can see clearly in our results here that it was authored by Kim. Just in taking that much of an approach on looking for things, you can see that we’ve really been able to take a machete and cut away some of that underbrush that we frequently get when we’re looking for search results.

Let’s take this a step further now and look at how else we might use this. Let’s say that I’m in search of something related to tasks. Now, if I just type in the word tasks, I’m going to get a lot of different things. Depending on what I was looking for, it could just be that I’m looking for something related to documents that have the word tasks in it. Or, it could be a classic problem that we have in SharePoint where I’ve got 15 different sites and each one of them has a task list and I’m just trying to figure out which lists have an assignment for me. Let me see if I can tackle both of those for you.

For the first one, let’s suppose that I truly do want to look for documents that contain the word tasks. Notice what I can do. I can put in this document and in this case, equals one means that I want it to be a document, equals zero means I want it to be a non-document. Just by adding the equals one, notice now I went from dozens of results, several pages worth down to just three. One little qualification on this one to be aware of as well and that is, notice the definition of a document. Not only do I get a Word document which is what a lot of us think of when we think of documents, we get library pages here. In the SharePoint sense of the word, they are in a library called site pages and they are in the documents.

Then, let’s take it one step further. Let’s say that we are truly interested just in Word documents. We could just go ahead and qualify the file extension on this just a little bit, and that would help us reign this in and see just the things we’re looking for. Right here I’ll just say, file type: docx. Then I’ll go ahead and search again. You can see now I’ve been able to refine it down to just the exact thing that I’m looking for.

Let me go one more step. Now I promised you that we would not only be able to find documents that have the word tasks in it, but that we’d also be able to find our task. In fact, I would tell you that of all the demos I think I’m going to do for you today, this is the one that is the most relevant and the most useful. That is, being able to find all of the tasks that are assigned to you across numerous sites. Let me start my query again. What I’m going to do now is, I’m going to change this one completely. I’m going to say content type:, excuse me, content class, and then STS_listitem_tasks.

Now, even if you are decidedly non-technical, this is a simple enough syntax that you can master it. Any time we see content class presented this way and then followed by some kind of qualifier, really what we’re saying is, “SharePoint, I only want you to bring back things of this kind.” In this case, we’re saying, “We only want you to bring list items that are actually of a task type.” Having typed that in, I’ll go ahead and search for that now. Notice, all I get are things from task lists. It’s pretty obvious that that’s the case.

Now, we could go a step further here and say, “I really want to find the ones that are assigned to me.” There’s sort of a good news, bad news going on here. The good news is, we can qualify who it’s assigned to. The bad news is that, there’s not an automatic way for us to say, “Only show me the ones that are assigned to the current user.” Those of you who’ve built these before and you know how you can filter a view where it says, “Just show items for me.” Unfortunately in the query language, we don’t have anything exactly like that. You are going to have to prompt your users for some kind of information. Or, it is possible with just a little bit of code, we could make this a fully automated process.

For this example, let’s just pretend that we are in the role of Kim. We’ll just come out here and say, “Assigned to” and “Kim.” What I’ve done now is make it possible for Kim to go to one place and see all of the tasks that are assigned to her. Again, she’s obviously not going to see any places that she’s not authorized to see. The thing here is, she’s now not going to have to go visit all these different sites to figure out if she’s got an assignment or not. She’ll just simply execute this query and she’s off and running. Now, how can we package this in such a way that it’s a bit more convenient to use? Well, let’s talk a little bit about that and then we’ll go a bit deeper and see how to get things done with that.

I’m going to switch back to Power Point here. The next part of this is called Accessing the Application. That is to say, once we figured out a way to ask for the results that we want, then we need a way to present it up to users so that they can conveniently consume it. Now, the easiest way to solve that problem is to recognize the fact that these search results can be expressed as a hyperlink. You might have noticed this when I was … demo, or you may have missed it, but if you take a look here, you can see. When I typed in all of this stuff in the search box, all of it’s represented up here on this URL. It is quite literally as simple as taking that URL and then making it available to a user.

How am I to do that? Well, I can put it on the Quick Launch menu. I could create a page and author a hyperlink there. Or, there’s even a hyperlinks list that’s available to me in SharePoint. I could put the information there. Pretty much anywhere we can store a clickable link, we’re now giving our users the ability to take advantage of this. Now, something that is a bit more sophisticated, but also brings us a bit more power and that is the Core Search Results Web Part. In particular, as developers or really high-end power users, we have the ability to customize the XSLT that is associated with that.

Now, if you have no idea what XSLT is, let me just take a quick aside here. Extensible Stylesheet Language Transformation is what it stands for. In very short terms, it’s just a template that describes how the results are going to look. As an end-user, you probably aren’t ready to engage trying to change that, but for a technical resource, it’s a pretty simple thing to do. Now, one other word about customizing the Search Results Web Part. The good news is, you can do it. You can absolutely do it. The challenge here is that, it can be difficult and if you’re doing this in a search center, you have to be aware of the fact that if someone else tries to modify that same set of XSLT, their changes might collide with your changes.

I want to drop that in your mind as a little bookmark of a problem, because when we start SharePoint 2013 conversation, we’re going to see that that problem gets solved. What I think is a better approach on this and it’s the one I’m going to demonstrate for you in just a moment is, when I’m building a search-driven app, instead of modifying the Web Part that’s in my search center on the results page, what I probably do and just did is just go to my site pages library. Create a dedicated page and put a Core Search Results Web Part on that. From there, I can configure the part so that it issues whatever query it needs to and it has whatever customization it needs. I am going to demonstrate that for you and let you see that in action. Let’s do that right now.

I’ll go back over and just continue working in the same demo environment that I’m in. Let’s do something a little more generic here. Excuse me. Let’s see if we can do a query that will show us all of the sites that someone has access to. Once again, I’m going to use the content class and I’m going to look for two content classes here. The first one I’m going to get is STS WEB and then I’m going to do an OR, and the OR does have to be capitalized. Then I’m going to do STS SITE.

Now, for those that are less technical in nature here, let me just give you a simple answer. If you want to get all the URLs, all of the sites, this is the way that you have to do it. For those of you that are a little more technically savvy when it comes to SharePoint, remember, if I did STS WEB, that’s only going to give me the sites. That’s not going to give me the top-level site for the site collection. For that, I have to use STS SITE. If I do those two together, you can see now I actually have a list of the sites that are available to me.

Now, in terms of making this a “application”, watch what I’m going to do. I’m going to copy this URL and then I’m going to navigate back over to the home page as my Acme Corporate Team Site. Then, once I’ve done that, I’m going to go in and customize this page. Once I’ve opened up page editor, I’m going to go in and insert a Web Part. What I’m going to insert is the links Web Part. I’ll put that right down here. Then, I’ll go ahead and save my work. Then once I’ve done that, I can go over to all site content and you can see that I have a links list here. I’ve got this one left over from a previous demo. I’ll just delete it out real quickly and then, add a new link here. I’ll put in the URL of the query that I just wrote and then I’ll give a description of all the sites you can access and click save.

Then once I’ve done that, now I can go back to the home page of this site and at any point, when a user comes to the home page of this site, they’d be able to click on this link and go directly to a search results page and see just the sites that they can access. To really drive home that this indeed doing what I’m claiming it’s doing, notice that I could take a second here and sign in as a different user. In this case, a user named Ed. Notice that if Ed clicks on this link, he’s only going to get two results whereas before I was logged in as an administrator and I got four results.

No worries here about writing any logic whatsoever, and we’ve solved that problem that we started talking about at the very beginning of the webinar. Now, to go a step further here, let’s see if I can log back in as administrator now and show you a little bit about how to make this a bit more beautiful. Okay, so I’ve logged back in as administrator. Now, let’s remember a point that I brought up early on in the process. I said that when we do these search-driven results, or these search-driven applications, it’s really nice to put the results in a format where they’re ready to use.

Saying that another way, if all I’m interested in are sites, then I don’t need all of this garbage. I just need a title of the site and something I can click on, and I don’t need the refiners and I don’t need all these other stuff. What I can do here is very simply create a page that contains a Core Search Results Part, and then configure that Part to issue this query. I’ve actually already pre-baked an example of that. Let me take you over and show you what that looks like.

In my site pages library, I actually have a page here called Available Sites. You can see that if I navigate to that page, it has a very simple list of the sites that are available to me. Then of course, I could click on any one of those and navigate directly to that site. Now, how is it that I’ve done this? Let me show you real quickly. I’m going to go ahead and put this page in edit mode. It’s being a little bit stubborn there. It doesn’t want to go in edit mode. Sometimes I have to switch back and forth between Internet Explorer and Chrome to get what I’m looking for. That’s fine. There we go. That’s behaving a little bit better.

Now, you’ll notice, the very first thing that I’ve got here is a Search Core Results Web Part. If I go in and actually edit this Web Part, I can show you the two things that I changed in order to make this thing work the way that I wanted it to work. Let’s drill down through some of these and have a look see here. Now, you can see there’s a section that describes my display properties. Then, you can see there’s a section where I have something to say about the results query. Let’s start with that one first.

Now, I know this text box is not big enough for you to see all of the contents, but you ought to be able to figure out pretty quickly what I’ve done there. In fact, if I select all of that and copy it, and then put it into Notepad so you can see it, that’s the exact same query that I issued before. Really, what I’m telling this Part is, the user is not going to type in a query. Instead, we’re just going to give them a fixed query, and then we’ll allow security trimming to make sure that everybody sees what they’re supposed to see.

Now, if I did only that, I would actually wind up with the same appearance and search results that I had before. Now, I want to go one step further and make it look nicer. I did that by adding my own custom XSL. Now, the point of this is not to teach you how to write this XSL. Some of you under the sound of my voice, probably would never do that, but the point here is just to let you know that it’s this simple. You can see in this care, really all I’ve done is just say, “Take all of the results. Turn each one of them into a hyperlink. Substitute the URL and then put a title on it.” That’s how I wind up with these very simple search results. Then, if I stop editing and go back into view mode, you can see that ultimately the page is just going to give me a very simple appearance and let me navigate to any one of those sites.

Then, you might have noticed when I was in the site page library, I also did a similar page here with tasks. Again, the only challenge on this one is, I’m going to have to do some kind of exercise to make sure that it works properly for different users. That’s probably going to involve a little bit of JavaScript. If you’re any kind of programmer at all, it’s not that challenging and it lets you do some very clever things. If you’re not already familiar with building those kinds of simple applications and letting the search engine do a lot of work for you, I hope that will inspire you to go forth and embrace some of these search-driven applications.

Now, as clever and cool as some of that stuff is, there comes a point where you say, “We want to do something with search that’s just a little more involved than a little bit of a power user trick.” That’s what this next part of the webinar is about. Now, one of the big ways that we can go a little above and beyond with search is integrating it with other sources. We have a couple of ways of doing that. One way to do that is to actually configure SharePoint so that it crawls external content. A classic example of this, maybe we have a line of business application where we keep data in a SQL server database. Among the things we keep in there, we keep our Part catalog information in there.

It’s actually possible to set-up SharePoint to use business connectivity services to connect into that database, to crawl the database and then to index a series of results that we can then search against. There’s a lot of good things that happen with that approach. We’ll talk about specific pros and cons on that in just a moment, but the idea here is that SharePoint is the thing that indexes and crawls the content and then any integration happens at that moment. At query time, in terms of getting the results back, it’s no different than any other SharePoint search.

Now, an alternative to that is something called Federation or federating the external content. In this example, the SharePoint search engine is not the one that’s doing the crawling and the indexing. That’s actually maintained by some external piece of software. Probably native to whatever repository we’re using. Then, instead of the integration happening at crawl time, it actually happens at query time. That is to say, when we do this kind of integration, even though they don’t realize it, the user is actually issuing two queries. They’re issuing a query in the SharePoint search engine and then on their behalf, it is issuing a query to this external engine.

What are the pros and cons of each approach? Well, first of all, the first one crawling external content. There’s several compelling reasons to do this. One, well, because it’s pre-indexed in the same way that other SharePoint data is, we just get better ranking of the content. In effect, it’s going to be subject to the same rules and the same ranking calculations that any other SharePoint data would be subject to. Another nice thing here, we have control on the frequency of crawl. As an example, let’s say that we are crawling some external data and we only need that data to be up to date within the 24-hour window. Well, we can save a lot of CPU cycles by crawling that information once a day as opposed to having some other mechanism crawl it on a more frequent basis.

This third one is particularly compelling. You’ll remember a moment ago, when I was able to do property specifiers in order to refine my search results, well if we’re crawling external data and we’re allowing SharePoint to do the crawling, we can actually specify those mappings. We can take bits of metadata from this remote source of data and bubble those up as properties that actually can be filtered in our queries. Then, in terms of maintaining the index, it’s just a single index for all of our data, and so our backup and restore operation on that is fairly simple.

The downside of crawling external content from SharePoint, well probably the biggest one would be the fact that we’ve got additional capacity requirements. Now SharePoint is going to have to have enough disk space to not only index its own content, but this external content as well. Not only is that a disk space equation, but also a CPU cycle equation. Beyond that, there is also this challenge that in some cases, we may be talking about external data that for security reasons, simply can’t be crawled. They may have blocked any external access in terms of being able to crawl through and read the data. That would drive us toward an approach where we would do federation instead.

The pros and cons here are mirror images of each other. In the case of federation, if we’re talking about a repository that already has a search engine, that search engine is probably already optimized for the kind of content that we’re searching. Rather than try to re-index it, we’ll just make a call out to the search engine at the query time and say, “Hey, tell us what you know about this.” This is particularly useful in cases where we simply cannot index the information externally because of security.

Then again, the cons here, well one of the biggest cons is that, we’re not going to be able to necessarily interleave the results on a single results page the way we would if we had SharePoint indexed it. More likely, you’re going to have a section of your search page that’ll show SharePoint results and then a section of your search page that will show external results. Not only that, when it comes to this kind of things, security trimming can be really tricky because at this point, you’re making that determination at run time.

Now, the good news, it can be done and in fact I can tell you that here at ThreeWill, we have written integrations on behalf of vendors where they took their products, integrated them on SharePoint and then when we did this run time federation or query time federation, we were actually able to make sure that the end-user still only saw the results that they were looking for. That is something that’s kind of a dark art. It’s not all that well-documented, but we’ve got a ton of experience with that and we’ve been able to help our customers get what they want there.

Now, just as before when I talked about building search-driven applications, not only can we make some trivial customizations to the Core Results Web Part, such as you saw a moment ago, but we can also do some pretty drastic customizations. This is another thing that we’ve actually done on behalf of one of our customers. I’m going to show you a pretty compelling demo of that in just a moment. Beyond that, if you’re familiar with search centers, we also have the ability to customize search center tabs that’ll let us do things like look for results in a narrowed results set or people searches or things such as that.

Then ultimately, if we just can’t get what we need out of the search infrastructure as it is, there’s always the option of writing either server code or JavaScript that makes a direct call into the search API, brings back the results we’re looking for and then we can specifically show that to our users. I can tell you again that, a particular ThreeWill use of this, we’ve actually written some application code for a customer where we did a typeahead search. Where as the user was typing in the characters of their search terms, we would go out and do a prefetch of matching items. That kind of thing is virtually difficult to do, or virtually impossible to do I should say in a search center, but can be done with custom pages and code.

Another thing that we’ve done on behalf of our customers, creating custom refiners. You’re probably already familiar with the concept that the SharePoint search engine gives you the ability to refine on different characteristics such as the date that the document was created or the format of the document. For certain custom data types, we can actually go in and create refiners from scratch.

Let me just show a little bit about how one might use this. What I’m showing you here is an example of an integration that we actually wrote on behalf of Polycom. You may or may not be aware of the fact that Polycom has a fairly sophisticated portal for keeping up with multimedia content. It’s called the Polycom Real Presence Media Manager. Particularly companies that do a lot of training videos or educational videos, video content of various types. This is a very popular piece of software that companies can use to manage that information.

Now, the product has a portal of its own for accessing that content and it’s really useful and it’s really nice. What Polycom has discovered is that, for a lot of companies, the destination of choice is going to be SharePoint. If we lock up all of our video content over in this vault where people have to visit another site, they’re not going to see it quite as often. Now, you can see on the screen here that we’ve actually created a Web Part that will let us go in and discover that content. I can look at things like recently viewed and so on.

Obviously, the demo that we’re talking about here and we’re more interested in here is this notion of search. Not only have we built a facility for browsing that content, we’ve actually integrated it in search. In this example, what we’ve done is, we’ve changed the XSLT associated with the Core Search Results Web Part so that we can not only interleave the data that’s coming back from Polycom, but also show it in an enhanced way. Let me show you what I mean by that. I happen to know as an example, that there are several videos and even some other information about Stroke. Again, obviously, the sample app here, being a healthcare app.

Now, notice you see some of the kinds of things you would expect. You see a title and you see some kind of icon, but you’ll notice that the information down here looks a little bit atypical. Not only do we have some author information and a date, we even have the duration of the clip. Then, this little link of Video Program Details. This is all custom code that we wrote on behalf of the customer. You can see the compelling story here is that, with one click, we’re able to preview that video in line and then if we so choose, we can even begin viewing it right there on the spot.

I think this is a pretty high-end example of how you can take the out-of-the-box SharePoint search bits, and then do some pretty significant customizations along with that. You’ll also note that not only did we do customizations related to the viewing of the content, but also refiners. Even though this is video content that’s held on another site and there’s really nothing SharePoint-y about it, we were able to supplement this information in such a way that we can refine it based on where it came from, who the author was and other such things. Again, the goal here isn’t so much to teach you how to do that. Obviously, that’s a pretty deep process, but rather just to alert you to the idea that it’s possible and it’s something that we have the experience and the expertise to do.

All right, we’re getting pretty late into our hour here. I’ve got one more section of content that I want to talk you through here, and that is just giving you a preview of coming attractions when it comes to Search in SharePoint 2013. I know that’s probably an over-the-horizon thing for some of you, but it is nice to know what’s coming and what’s to be expected.

Probably the biggest difference when it comes to the architecture of Search in SharePoint 2013, and that is this notion that we now have a single search engine. For those of you with a lot of deep knowledge on this, you know in the past, you’ve got SharePoint Search and FAST, and different versions of those. Now it’s just a single unified product. Really, I think that’s a good thing for the most part in that, it reduces some confusion. When you’re authoring queries now, you don’t necessarily have to be cognizant of, “Well, gee, which version of the search engine am I on?”, in order to be able to do this thing that I want to do.

Now, in terms of the languages that you use or the query languages that you use, keyword query language, which is what I’ve been demonstration for you today is obviously still supported. In addition, in prior versions, FAST had its own unique query language. Even though FAST doesn’t exist as a separate engine anymore, the FAST query language is still supported. The one that isn’t and it isn’t still supported is the SQL variant of being able to do search queries with a SQL LIKE Syntax. I think at this point, that’s one that didn’t get used a whole lot and that’s why Microsoft made the choice to drop that. If you’re under the sound of my voice and you’ve got some things that you’re doing in SharePoint that are dependent upon that dialect, then just be aware of that. That’s something that’s going to go away when you get to SharePoint 2013.

Some other things and some of these are technically meaty. Others of them are more marketing points, but they’re worth making. There’s some improvements to the results ranking model. Another one that I think is really cool, and that is the results are further refined based on click-throughs when the results came up before. Those results that have a higher rate of click-through, will show up as being more important on subsequent queries.

Another thing that’s new, a moment ago, I’ve talked to you a little bit about federating outside sources of data. That’s gotten a new name now. It’s called Result Sources. It’s really a way of just describing external data. Probably the biggest thing here that would be of use to developers is the fact that the way that we do security trimming here is a little more elegant and a little more sophisticated. In terms of being able to write customizations, that’s a good thing.

Now some other things, some refinements and changes to query rules. We have this thing called Promoted Results. This was formerly known as Best Bets. We’ve got a little more flexibility on how that we use that. We have this other thing called Result Blocks, and I’m actually going to talk about this one a little bit more on a later slide. The idea with result blocks is that you can use it to group related results. As an example, let’s suppose that somebody searched for a particular kind of content and maybe we discovered that based on certain words. Probably they’re looking to highlight some things related to Power Point presentations. Well, we could actually set up a separate result block that says, in addition to all the results we’d show them, we could actually use a result block to call out just the Power Point presentations that are in those search results.

Again, if you think about where you visited commercial search engines like Bing or Google, and they often have a way of saying, “Well, gee, you searched for this thing, but there’s a whole bunch of images that match. Would you like to see those?” That’s exactly what result blocks are all about. Another thing that’s kind of cool is that, you can actually boost rank results based on certain terms. As an example, suppose that someone goes in and types in a term to download a utility, we can actually configure a query rule that says download is recognized as an action term and based on that, if we have a particular site in the company where people are supposed to be getting their downloads from, we can actually boost that result and that would result in a likelihood that it would be higher in the result stack.

Another nice innovation here, Continuous Crawl. Again, those of you that have worked with SharePoint a long time, you know that there’s always this challenge of, you want to set your crawls frequently enough to be up to date, but not so frequently that they killed the server. Now we have this concept that, for particular sources of data where it absolutely has to be up to date, continuous crawl really is just tapping into the transaction engine from the underlying database and saying, “Anytime we have new content, we’re just going to go ahead and update the sear- index immediately.”

Now, let’s talk for a moment about a hybrid environment. This is something that I think a lot of you, particularly if you work for big companies, you’re going to encounter over the next year or two. It’s interesting not only to identify it, but to talk about some of the search implications of that. When I say hybrid environment, what do I mean? Well, we all know that Microsoft is making a big push to move customers to the cloud, to begin using SharePoint Online as a part of the Office 365 Suite.

Now, for most companies that have a significant amount of content, it’s probably not going to be possible to move exclusively to the cloud in a short-term. There’s really two big reasons for that. Number one, they’ve probably got customizations that wouldn’t run in the cloud particularly if they’re farm solutions. Number two, it’s probably not practical to take the gigabytes of data that they already have in place and simply move those up to the cloud in order to sit there. It would be very expensive to move them up there and it would be an expense to have it up there in an ongoing basis.

In order to combat those barriers, a lot of customers are using what’s called a hybrid environment. In simple terms, they’re going to take everything that’s on-premise and for the most part leave it there. Then, new content is going to be put into the cloud and used there. Now, the presumption is, if we are going to have such an environment, we really need to be able to integrate the search results from those two environments. To say that a little bit differently, we don’t want users to have to constantly say, “Well now, wait a minute. Is this thing in the cloud or is this thing on-premise?”

The idea here is that, we can use the result sources feature that I mentioned just a moment ago in order to integrate those results. How are we going to use that? Well, suppose that we’ve got an on-premise search page. What we can do is create a result sources feature and then a result block that highlights the matching cloud results. They may not be interleaved one for one, but what we’ll see is a section of highlighted results. Then, if someone observes, “Well, gee, the results that I think are relevant to me, there’s more of them on the cloud than there are on my on-premise.” They can actually click on one or more items inside of that result block and it’ll navigate them to the cloud. You can also do the same thing in reverse. You could set up a cloud search page so that it not only shows cloud results, but also has a results block that highlights on-premise.

I can tell you that we are actually doing a proof of concept of this as I speak for a company here in Atlanta. Fairly large company that you would recognize and we’re getting some really good results with that. Really kind of two good news fits there. Number one, it’s doable and it works. Number two, when you find yourself at a point where your company needs to do that, we’ve got the expertise to help you get that done.

Some other things you’ll want to watch for in SharePoint 2013. Some of these are just features that we’re going to like and take advantage of. Some of them are things where we actually have opportunities to customize. We now have this concept of a hover panel for particular content types that show up in search. You can actually hover over those and preview the results on a mouse over. We also have the ability to create our own hover panels. If I were to go back to what I showed you a moment ago on the Polycom integration, once that products gets updated to SharePoint 2013, it would be a doable thing for us to go in and write a hover panel where instead of having to click to view the video, one could simply float the mouse over and see a preview of the video playing. I think that’s a pretty cool one.

Another nice thing of SharePoint 2013 Search, if you look at the search pages I’ve been showing you up to this point today, the icons and the graphics associated with the results are pretty bland. They don’t really communicate a lot of information because they’re tiny and somewhat nondescript. Microsoft has tried really hard to put some good visual cues on your search results so that it’s real obvious, “Oh, that one’s a Word doc, and that one’s a diagram and that one’s from Visio and another one is an image.” I think that is a much needed improvement.

Some other things. I know that not all of you take advantage of the term store and the managed metadata capabilities of SharePoint 2010, but if you do, there are some greatly improved capabilities there in terms of being able to navigate and find items. Another improvement that we have, this concept of display templates. Now, you’ll remember that I’ve talked twice now about changing the XSLT in the Core Results Search Web Part either on a particular page or in a search center. Given the difficulties associated with that, it’s nice now that we have an alternative to that.

Really, what are we saying when we say display templates? Now we’re able to build what are in effect, customizable little nuggets of HTML. We’re going to be able to associate these in such a way that when a particular result set comes up, we can actually modify how it’s going to look on the results page. The nice thing about this is that, because it is associated with particular types of content, we can do it in a search center and do it in such a way that we won’t collide with other customizations. It’s more flexible, it’s more powerful, it’s less error-prone, it’s easier to do. That’s nothing but a win when it comes to doing this.

Now, the various parts that make up this, there’s really two things. First of all, you have a controlled template that’s going to describe the header and the footer and the layout of the results list. Then, an item template that we’ll use to describe the presentation of the items in that list. Those of you that do a little bit better with pictures instead of words here, this gives you an architectural representation of what we see. Then, this is a nice screenshot that shows you how you could do a search results page with an image slider. You’re able to just simply scroll back and forth through the images used and a little bit of jQuery in addition to the HTML template to show this.

If you are in a place where you do customization of search, your world just got a lot bigger. Conversely, if you’re a consumer and you need these kind of capabilities, it’s nice to know that it’s finally available in the product, and that you’ve got some folks in ThreeWill that know how to make that happen for you. Let’s talk about some takeaways real quickly, we’re getting close to the top of the hour here.

First and foremost, if you’re a developer and you’re under the sound of my voice, you need to start thinking about search-driven applications. If somebody describes a business problem to you and the answering of that business problem can be done from the search engine, it’s faster, it’s more efficient, it’s easy to write, there’s a lot of wins there. Obviously, if you’re doing a traditional data-entry app where you need to add items to a list, that’s not necessarily a search-driven application. Anything that involves asking questions. This is a good candidate.

Another thing to think about, beyond just the building of these simple search applications, if your company is in a place to make a higher technical commitment, there are a lot of other things that we can do. Customizing the search results page, writing refiners, integrating external data. Another thing to look forward to, SharePoint 2013 really offers us some much better opportunities to integrate at a deeper level. Better security trimming, better models for extending the UI, and doing those things in such a way where it’s not going to be brittle or prone to breakage if somebody else tries to do a customization as well.

Then, obviously, what I want you to take away from that is, this is what we do. It’s not that we’re going to do these things. We have already done these things on behalf of our customers, and we’ve had some really good results with that, and we want to help you get those same results as well.

read more
John UnderwoodEnterprise SharePoint Search Webinar

A Tasking Exercise – Working with SharePoint Tasks

John Underwood is a Technical Evangelist at ThreeWill. He has nearly thirty years of software development experience. He is an experienced technical instructor with superior presentation skills and is proficient in delivering standard curriculum as well as developing and delivering custom curriculum.


One of the interesting features of SharePoint 2013 is the introduction of the My Tasks feature.

It addresses one of the more common problems with SharePoint: you’ve been assigned various tasks across many different SharePoint sites, and now you have to keep up with all of them. My Tasks consolidates the tasks assigned to you in one simple view and saves you the tedium of visiting each site containing assigned tasks.

So, what if you’re in a company that is slow to upgrade to SharePoint 2013?

Well, it turns out there’s a clever search technique that you can implement in SharePoint 2010 that will give you similar results.


How many times have you asked a question and gotten an unsatisfactory answer, only to rephrase the question and get what you’re looking for?

Well, that can also be the case with SharePoint. We can use the SharePoint search facility to locate tasks by simply using the query term “tasks”. However, doing so will probably produce results that are too broad:

Everything that has the word  "tasks" in it!

Everything that has the word “tasks” in it!

Now let’s ask the question again, but be a bit more specific about what we want using the following query:


Notice that this produces a startlingly different result…

It only shows tasks!

It only shows tasks!

Depending on your technical background, this query either makes total sense or looks like a foreign language. Either way, at its core it is a simple request: “SharePoint, please only show me items in a task list, and nothing else.”


Great, we’ve narrowed our search to tasks only. But the real goal was to find only the tasks assigned to a certain person (that person being “me”, of course).

It would be nice at this point if SharePoint included a keyword that refers to the current user (such as [Me] when creating a view) but sadly that is not the case. However, it’s easy enough for us to ask for items assigned to a particular person using the following query (in this case we’re assuming the user’s name is Jim Shorts):

ContentClass:STS_ListItem_Tasks assignedto:"jim shorts"

Simply put, “SharePoint, only show me tasks, and only show me the ones assigned to Jim Shorts.”

Notice Jim has tasks on two different sites.

Notice Jim has tasks on two different sites.

Where do we go from here?

If you look carefully, you’ll notice that the end of the address for the results page for this search looks something like this:


Simply put, the query details get appended into the URL. Now then, how do you run this query again to see your latest tasks? Well, just add the URL to your browser’s favorites list and then click the link anytime you want to see your latest tasks.



Want to learn more about what SharePoint Search and search-driven applications can do for you?

If so, please attend our upcoming webinar entitled Enterprise SharePoint Search. You’ll see even more about search-driven applications as well as learn about how ThreeWill can help you get the most out of SharePoint search.

Register Now
read more
John UnderwoodA Tasking Exercise – Working with SharePoint Tasks

ContentType SQL Query Tip

Kirk Liemohn is a Principal Software Engineer at ThreeWill. He has over 20 years of software development experience with most of that time spent in software consulting.

Whenever I used ContentType in the WHERE clause of a FullTextSQLQuery the query would return no results. However, I found a fix:

  1. Go to Central Admin > Search Service Application > Metadata Properties.
  2. Edit ContentType and checked “Allow this property to be used in scopes”.
  3. Do a full crawl.

In some tests I reset the index, but that does not appear to be required (I did one test without resetting the index).

Note that this is not required for many of the other managed properties if you want to use them in the WHERE clause.

FWIW, I looked at the content type properties in PowerShell. Checking the checkbox changed the two properties in red below.

PS C:\Users\Administrator> $ctype = Get-SPEnterpriseSearchMetadataManagedProperty -SearchApplication “Search Service Application” -Identity ContentType

PS C:\Users\Administrator> $ctype | fl

ID : ContentType

PID : 60

Name : ContentType

SplitStringCharacters :

SplitStringOnSpace : False

Description :

ManagedType : Text

FullTextQueriable : True

Retrievable : False

UseAAMMapping : False

MappingDisallowed : True

DeleteDisallowed : True

QueryIndependentRankCustomizationDisallowed : True

EnabledForScoping : True

NameNormalized : False

RespectPriority : False

RemoveDuplicates : True

HasMultipleValues : False

OverrideValueOfHasMultipleValues : True

IsInDocProps : True

IncludeInMd5 : False

NoWordBreaker : False

UserFlags : 0

Weight : 0

LengthNormalization : 0

EnabledForQueryIndependentRank : False

DefaultForQueryIndependentRank : 0

IsInFixedColumnOptimizedResults : False

RetrievableForResultsOnly : False

PutInPropertyBlob : False

QueryPropertyBlob : False

UsePronunciationString : False

MaxCharactersInPropertyStoreIndex : 64

MaxCharactersInPropertyStoreNonIndex : 4000

EqualityMatchOnly : False

MaxCharactersInPropertyStoreForRetrieval : 65536

DecimalPlaces : 3

IsBacked : True

The blog that led me to the solution is here: My issue occurred with only a couple of conditions and not 10+ conditions as mentioned in this blog. In addition, I am using SharePoint 2010 and not SharePoint 2007. It was the same fix for both scenarios nevertheless.

While researching this I found several posts about ContentType where people have a hard time selecting the ContentType in the full text SQL query. Using ContentType in scopes does not help with that problem. The solution others have found is to create your own metadata property and have it map to the same crawled properties that ContentType maps to. Here are a couple of posts that mention this:

read more
Kirk LiemohnContentType SQL Query Tip

Custom Search Refiners

Will Holland is a Senior Software Engineer at ThreeWill. Will has proven to be adept at understanding a client’s needs and matching them with the appropriate solution. Recently he’s developed a passion for working with .NET, MVC, and cloud-based solutions such as Microsoft Azure and Office 365.

Default refiners are great….if you have some pre-existing knowledge of the document.

Out of the box, SharePoint’s search feature makes finding documents, videos, and other items much easier when the user doesn’t know where to begin looking. Sometimes, however, it’s just not enough. If your site collection contains a large number of documents, as was the case with a recent project I was on, a user’s search may return hundreds of results. The only methods the user has at their immediate disposal for narrowing those results down is to either search again with more keywords or use some crummy filters provided by the Search Refinement Panel. The default refiners, such as author, require that the searching user know details of the document prior to searching for it. Great if you just published a document and immediately had a brain fart; not so useful for the rest of us though.

Fortunately, it’s not all that difficult to create your own custom refiners that will provide your users with the experience they have been searching for. All puns aside, adding custom refiners will allow your user to narrow their search results down by more meaningful filters, thereby reducing the amount of time spent searching and, hopefully, increase productivity.


  • You have an existing site collection with the Search feature active and configured.
  • You have sufficient permissions to access and modify Central Administration.
  • You have permissions to edit pages and web parts.


For the sake of consistency, I’ll be using the “Productivity Hub” as my example site collection. The Productivity Hub is a SharePoint 2010 site collection that contains a plethora of content for users making the migration from Office 2007 to Office 2010. It’s a free download, available at

The content for the Productivity Hub is stored in a document library, aptly named “Content Library”. The content library contains a “Product” column which indicates what Office product the document is intended for. This is the column that I will use during this tutorial.

In the beginning…

To start things off, the first thing we must do is determine what data we have that could be useful to searching users. Something that helps them define what they’re looking for. If your site collection contains a large number of documents that relate to specific clients, it would probably be useful if the user could narrow their search by documents that only pertained to a chosen customer. In my case, my documents all pertain to various products in Microsoft Office 2010, so I’ll use the “Product” column so that my users can narrow their search results down based on specific Office products.

As a best practice, the best columns to use for metadata are those that are indicative of the document contents, which is why I’m not a huge fan of the OOB search refiners. It’s also best if the data stored in the column is NOT a multi-value choice or multi-line text.

If your library doesn’t contain a particularly useful column…create one!

If you must walk before you run, before you walk you must have crawled.

Unless you created a new column for your library, the chances are that your desired column has already been indexed when the last scheduled site crawl ran. Regardless, we still need to verify that it was crawled. You know what they say happens when you assume something.

The Crawled Properties Page of Central Administration

  • Open up Central Administration and navigate to Application Management | Manage Service Applications
  • From list of Service Applications click Search Service Application
  • From the Queries and Results section, click the Metadata Properties link.
  • Finally, click the Crawled Properties link.

This page contains a list of every column captured during a crawl and, hopefully, your column will be in this list somewhere. To find out, type the name of the column in the search bar towards the top of the page and let’er rip. With any luck, you’ll receive a few results. My search for “Product” resulted in quite a few, but the one I’m interested in is ows_Product(Text). If given the choice, the crawled property that contains the ows_ prefix is usually the one you want.

If you created your own column or your search results didn’t include an ows_{YourColumnName}({YourColumnType}) type result, then I’m sorry to say that before you can continue, you’re going to have to do a full or incremental crawl. This may or may not be an issue for you depending on how long a crawl takes to complete.

Once you have verified that your column has been crawled, click the Managed Properties link followed by the New Managed Property link.

The “New Managed Property” window.

  • Fill out this new window by giving your property a useful name.
    • Try keeping the name to one word, but if you need to use multiple words be sure to use Pascal casing.
  • Give your property a useful description.
  • Select the appropriate type.
  • In the second section, select Include values from a single crawled property based on the order specified.
  • Click the Add Mapping button.
  • Type the name of your crawled property and click the Find button.
  • Select your crawled property from the list and click the OK button.
  • Everything else can be left at its default values, so click the OK button.

Setup for Search Success

With our new managed property in place, we’re ready to begin adding our refiners.

Navigate your way to your search results page and put it in edit mode. The first thing we need to do is include our new metadata in the search results.

  • Locate the Search Core Results Web Part, located roughly in the middle of the page.
  • From the web part context menu, click Edit Web Part.
  • In the web part editor, expand the Display Properties section.
  • In the Fetched Properties text box, scroll towards the end until your cursor is directly in front of the </Columns> tag.
  • Add the following: <Column Name=”ProductName”/>, replace ProductName with whatever you named your managed property.
  • Click the Apply button.
  • Click the OK button.

Next, we need to define our actual refiner.

  • Locate the Search Refinement Panel web part, located on the left side of the page.
  • From the web part context menu, click Edit Web Part
  • In the web part editor, expand the Refinement section.

We need to add our refiner definition in the Filter Category Definition text box. This text box is quite a bit wordy, so it may be easier for you to copy the contents and paste them in an editor that will format it for you, such as NotePad++ or Visual Studio.

I do declare!

Once you have your code formatted, you can see that it’s an XML document that contains a series of <Category> definitions. These definitions are what define the refiners that appear on the refinement panel web part. You may also notice that they appear on the web part in the order that they are defined here, so a decision must be made on where you want your refiner to appear. I think the Product refiner is pretty useful, so I’ll simply insert my code before the other categories. Below is the definition I’ll use to declare my refiner.

Description="The Product associated with the item"
MetadataThreshold="1" NumberOfFiltersToDisplay="5"
MaxNumberOfFilters="0" SortBy="Name"
SortDirection="Ascending" SortByForMoreFilters="Name"
SortDirectionForMoreFilters="Descending" ShowMoreLink="True"
MappedProperty="ProductName" MoreLinkText="show more"
LessLinkText="show fewer">
ValueReference="Absolute" ShowAllInMore="False">








As you can see, the Category tag contains a large number of properties that you can configure. Here are a few that you’ll certainly want to understand.

  • Title – This is the display name for your refiner category.
  • MappedProperty – Extremely important. This should match the name of your managed property and the name specified when you added the column tag to the search core results web part. It is case sensitive!
  • MetadataThreshold – The number you specify is how many search results are required to be returned that match a filter before it will be presented to the user.
  • NumberOfFilterToDisplay – The number of filters to be displayed before the user has to click the “show more” link.
  • MaxNumberOfFilters – If the user clicks the “show more” link, how many filters do you want them to see. Set the value to zero if you want to show all filters.

For the complete list, please refer to this MSDN article: Refinement Panel XML Schema

The CustomFilters element has a few properties of its own, but for the most part, it will use the values shown above. The most common exception to that is when you’re refining by dates. The link above provides an explanation for each of these properties as well.

Finally, you have the CustomFilter and OriginalValue nodes. The “Name” property of CustomFilter is the text that will be displayed on the refinement panel, so keep it short and precise. The value between the OriginalValue tags is the value we expect for each custom filter. Again, this value is case sensitive. 

Once you’ve created your declaration, copy the contents of your editor and find your way back to that cluttered text box. Replace its contents with your code.

Click the Apply button and stop editing the page.

And you’re done! Well, almost. If you do a search, you notice that your refiners aren’t showing up.

I just learned to run, now you tell me to crawl again?

Yup. You’ll have to do another full crawl before your refiner will appear. Pretty much anytime you make a change to the search metadata, you’ll be forced to do this.

However, once that’s completed….

Ta-Da! My Custom Refiner LIVES!

read more
William HollandCustom Search Refiners