Chris is a Senior Software Engineer at ThreeWill. His area of focus is consulting in the development of Microsoft .NET and SharePoint technologies. His primary role has been development lead in recent projects. Project roles have ranged from Development/Technical Lead to Development Resource.
Today, I would like to take a moment and walk you through, at a high level, what a Jive to SharePoint migration process looks like.
For the migration process itself typically we would recommend that you have a couple of servers or a couple VM’s. It’s not mandatory that you have a couple of servers. 1 server or VM will work just fine. Definitely recommend at least a couple of processors because what you do is parallel processing to improve the overall performance of the tool, but that’s just the recommendation. Also, basically having a Jive account that has read access to everything and a decent network connection. You going to be hitting Jive pretty hard and pushing things into SharePoint, so you will need to have some decent throughput for that.
You will need a fairly large hard disk or hard drive space so that you can store all the physical binaries that you will be pulling out of Jive. I’ll explain that here in just a minute. The first major thing we do, moving into the process, is to capture the inventory of content from Jive. The idea is that we want to pull the content from Jive into our local SQL Server database and partially into the file system so any of the binaries, images, Word documents, and things like that all get pulled into the file system. Everything that is metadata wise and everything that connects the pieces together gets pulled into a sequel database. The inventorying process basically does three primary operations.
We will do these in sequence. We do a command, what’s called get people, which means we pull all of the individual profiles and people references from Jive into our database. Get places, actually pulls all of the individual spaces, groups, projects, and blogs. Place is just a generic term to reference all four of those commands but we pull all the hierarchy into our database, so we know a complete list of all the different places that are in Jive. Next, we do what’s called a shallow pool of gathering content from all the places that we pooled into the database. Shallow basically means we’re not pulling all the binaries and we’re not going any deeper into what the API Jive has allowed us to see. We’re creating a high-level idea of the content count.
Moving on from there, we go in and we look at the output, we produce a spreadsheet typically as a result of this inventory phase and we transition into the para phase where we actually go in and produce a spreadsheet. We go through and define, what are the SharePoint structure? What should it look like based on what we see in the spreadsheet? Do we need to provision any site collections? Are there any managed pass we need to be aware of? Do the basic design of what the SharePoint environment’s going to look like and really leveraging the inventory that we pulled in the previous step. From there, if we’ve got a substantial amount of places to migrate, we typically break things down into some logical groups. If we have a large amount, it’s typically that we do it in groups of 1,000 and work with that at a time.
Then that’s the breakdown of where we were from a client perspective. Internally, a lot of time we’ll work with batches of 50 at a time so we can better manage what we’re actually pulling out of Jive, what we’re actually transforming, what we’re actually pushing into SharePoint, and be able to validate, get a little smaller chunk. Overall, though, we try to do things at a large level like 1,000 at a time.
Then we move into more of the migration phase where we’ve got everything planned, we pulled out initial content, now we need to go through and there are essentially 4 different commands we do for each of the batches and that batches of 50 is what applies here. We validate and if a SharePoint site is not created, we validate and see that it exists. If it doesn’t exist, we have the ability to create that on the fly.
If needed, we need to recreate the SharePoint permissions. Probably won’t get too deep into that in this particular demo but we can definitely talk about if that is something that’s important to you. Get content is also run here. You saw it was run at inventory but it’s also run here again where we actually do a deep pull, so not a shallow pull. We’re doing a deep pull this time to pull the actual content and binaries and everything into the database and file system. Kind of some new commands we’ve got here. Transform content basically what that does, it takes the content we’ve pulled out of Jive and it makes it ready for SharePoint so think of all the URLs, things that pointed to between documents, between places, things may point to Jive, things may point to a user profile in Jive. We take it, scrub, and change all that so that it actually references SharePoint and the SharePoint equivalent.
That’s what goes on in the transformed piece. Make sure everything is SharePoint ready. Then the uploaded content is the final command that says, “Okay, everything that you’ve transformed and you’ve prepared for, let’s go ahead and upload that to the target SharePoint file,” and that’s really what completes the migration.
Then really once we’ve done that, we’ve done our batch of 50 let’s say, we’ve gone through this process, then we go through a validation phase and we check a certain number. We validate counts to make sure things look … With the same count that was in Jive and do we have that in SharePoint, we do some basic inspection to make sure links and things are working properly, and then we have some high-level checkpoints to go through before we transition to the customer to do a deeper inspection. That’s the high-level process.
Really what we’re going to dive into is really the migrating stage. I’ll show you a little bit of the prepare in the batch portion of it but we’ll dive into the migrate so you can see the commands, how they execute. I don’t want to bore you with the inventory process because it takes a while to run some of these and not really conducive to a good demo. That’s basically the high-level process. Hopefully, that makes sense.
Just real quick, another follow up to the inventory process just so you guys can see the database that we were talking about earlier. Typically we create a Jive to SharePoint database. The first time the tool or utility is run, it will actually create this database for us so as long as we have a valid connection string, it knows how to create the database and the structure. I think I mentioned here earlier that we do a get people, get places, and a get content so just to translate that for you guys, if you come into the database and you’ll see a multiple set of tables here. If you go into the Jive persons, that equates to the get people so if I do a quick-select on that, I can see that I’ve got in this case I’m running against the Jive sandbox which is just a generic container or generic Jive environment.
In this case, I’ve got 561 people that I was able to find in the Jive sandbox so that’s the result of getting people, it produces this structure. The same thing forgets places. If you go to the Jive places, I pulled 1,000 rows there and you’ll see a list of all the different places and if you look here, you’ll see blogs, projects, you’ll see groups, you’ll see spaces so those are the 4 different content types that really make up a place. Here’s all the collection of places we found. Actually found over 1,000 because that is what we’ve limited it to. Let’s see what we actually have here. 1,939 places. That’s what we found in the Jive environment.
Then the next level would be going to Jive content and if I select from that, you’ll see that there’s content here as well. One of the things you’ll notice, you’ll see content and then transformed content. When we actually do the real transform, I think I mentioned to you earlier, that is the process that takes the Jive content, scrubs it, makes it ready for SharePoint so you can see that some of these have already been done. We actually store this transformed content into this particular table as well so that’s that transition from inventory into the actual migration itself. I’m going to stop here and we’ll pick back up with the physical migration.
Okay, so as we go back to our diagram, we’re getting ready to prepare and then create the batches for our particular migration. In this case, we think we’re just going to migrate a couple of places just to keep it simple. What I’ll do is I showed you the database. What we do is we actually have a query we run and we produce a resulting spreadsheet from that database. This is where we would handoff to the customer or actually work with the customer to go through their inventory of all the different places. This is very similar to what I just showed you in the database with the Jive places table. It’s got all the place ID’s, it’s got all the place names, everything is in here. The types. Basically a full list of places as well as content counts. How much content is associated with each of the different places based on the types?
Not a whole lot showing up here just because it is the Jive sandbox environment. If I go off to the right, in fact, let me do this real quick, I’ve actually got a couple of places in here that I’ve set up to be the ones we want to migrate. In this case, I said, “Let’s go ahead and make these our pilot sites,” so a lot of times when we do a migration, we’ll do maybe 10 pilot sites. In this case, we’ll just do a couple. I’m doing the Lisa test group 3 and then the Lisa test group 3 blog that lives underneath that particular place.
This is resulting or this is the source Jive environment that we’re going to be looking at. If I come over here, the reason I wanted to show you this is that this is the spreadsheet that we work with the customer to determine where things are going to go into SharePoint so what I’m doing is I’m saying, “I’m going to use the site’s manage path,” just a standard manage path in SharePoint. We can use custom ones if you’d like. A destination site, in this case, it’s my dev site and then I’ve got a structure that I want to use underneath this, current, and then another site underneath that. Lisa test group and the Lisa test group blog so we’ll see how the tool will actually use this information and create these as needed and then actually migrate the content of these particular locations but these are the fields we would actually use and also in the database itself to direct the content, where how it goes from Jive to SharePoint. Here is our source Jive place that we’re going to be starting with, the Lisa test group 3.
Just has a bunch of just test content, things special characters. Just some basic content we’ve used to try different edge cases and really test the product itself but just wanted to show you that this … How the migration process works so it’s simple for that. There are 44 pieces of content here. That’s a mixture of Jive documents, documents like Word documents, discussions, there’s even a couple of videos in here. I believe there are blog posts so one of the distinctions here is that blog posts look like they live in the same place in Jive but they actually in reality live in a subsite so what we’re going to do is migrate this place into SharePoint but then we’ll also migrate the corresponding blog and blog post into SharePoint as a separate sub-site of a … Called a blog so we’ll look at that here in just a minute but just wanted you to see the structure, how things are in here, and how things are going to look when we go into the SharePoint environment.
Let’s just pick one so we can have something to reference. Lisa’s awesome discussion. Basic details, here are some headers, some images, some text. We’ll look at the same sort of thing in SharePoint once we’ve done the migration. Now we’ve got the commands up here. Our little cheat sheet of commands for doing a migration for 2 particular places so we’re going to walk through these and I’ll show you how to execute them on the command line itself. I’m going to go ahead and kick this first one-off and then I’ll explain what it does.
The first one we’re doing is called validate sites and you can see here, they all start with the same sort of ideas that J2SP, Jive to SharePoint. Dash A says action and validates sites. Validate sites, what it does is it says based on these 2 different places, these place ids. Dash I’s are the place ID’s, go ahead and verify to these places based upon our mapping that we talked about earlier, are these places already set up in SharePoint. If they’re not and we have the dash C option, it’ll go ahead and create the places for us so let’s take a look at our log file and see what it did. All right, so we always want to look for truth in the log file for this particular command. We can see here it’s got the dev, C Edwards. Found that, return true. Then either found or created the dev CA words current AA Lisa test group 3 and the same thing for the blog. Return true in both cases. We found our items, we found our places, which is great. That means we want to move on to the next command, which is to get content.
We talked about getting content earlier with an inventory. We kicked this off and I’ll explain what it’s doing. Get content in this case, we’re doing something a little bit different. We’re doing a dash B for binary so we’re pulling the binaries. We’re going deep which basically means if we need to pull categories or other things that are related to the content, which requires additional API calls, we do those as well. The dash F says to do a full pull so anything that’s changed since it basically doesn’t matter if it’s changed or not, just pull everything again. Dash P says run parallel so if there are multiple processors in the machine, we can actually take advantage of that, and then the dash I for the place ID’s is consistent through all those commands since these are the places we want to pull and then the last part of it, I don’t know if I mentioned earlier is basically just a pipe to a log file so we can actually see what the output was of this command. Actually, if you look at this you can see that it pulled a bunch of content here.
I won’t bore you with all that but it took 29 seconds basically to pull the content for those 2 places so not too bad. Let’s go ahead and run our transform content. Transform content doesn’t take all the command lines options, the other ones did but from above it, just basically takes a list of place ID’s, like I mentioned earlier, what transform content does is it takes the content we pulled from Jive whether it’s in the database or in the file system and makes things ready for SharePoint so it fixes profile URLs, it fixes any link to images, fix link to content, makes sure that everything is related to SharePoint and that’s all that’s addressed here. We’ll let this one run in the background.
Actually, while this is running, we haven’t actually uploaded content yet so this may take a moment to run because we’re still kind of haven’t pushed anything to SharePoint yet, let’s take a quick look at our SharePoint sites. If we come here, some sites. There’s our AA Lisa group, test group 3. We can see here that there’s really nothing in here yet. If we look at the content, there are 0 documents and really no content’s been uploaded. I can tell by looking at the menu but for you, you can tell there’s nothing in here. It did create the subsite for the blog. AA Lisa test group 3 so we go into that. You’ll see the blog site is being created. Just with the generic empty blog. If I look at the post, there should be nothing in here yet except for the initial blog that gets created when the site’s provisioned. That’s all that we want it to look like. Let’s go ahead and go back to our test group 3.
We’ll leave it set here until we come back to it. Let’s see how our transform’s doing. It looks like the transform finished so let’s go on to our next command, which is the … Oops we don’t need that. Let’s go into our next command, which is the uploaded content, and here’s where we physically begin pushing stuff into Share Point. I hope I pasted that properly. There we go. All right, so here’s where we’re actually physically uploading the content into Share Point and just to prove that, if I come back in here and I go to select contents, you’ll actually see some stuff changing. You can actually see a Wiki page, a Wiki library’s been created now, a discussion’s been created and things are getting uploaded as we talk. If we go back to home, the menu structure’s to be a little different. This will actually get changed at the end. It will update this but I can tell based on the fact that it shows the video as discussions, Wikis that it’s actually in progress of uploading content so that’s a great sign.
We can actually look at the upload. We can see that it’s doing some work. This might take a little bit of time. Okay, so we see now that the uploaded content’s finished. Let’s go ahead and take a look at the resulting content in our migrated site so if I come over here and I click on documents, the first thing you’ll notice is kind of our default, our standard way of transforming and uploading this content is to use a Wiki library, really because we can maintain the relationships that Jive has between containing content, content and then socialization settings like likes, things like that. We want to make sure all that stuff is maintained properly and Wiki libraries have been the best ways of doing that. There are other ways of doing it and we are not necessarily set on this. This is the default mechanism the tool uses is a Wiki library for collaborative documents, files, photos, things like that.
Let’s go ahead and jump into a document. You can see here, I’ve got some basic information about the document, in this case, has an attachment, so we maintain attachments. Describes what the document is. Again, there’s not a whole lot of real HTML rich content here so if you actually had rich content, you’d see it represented here in that sense. Let’s see if we can find some stuff that has a little bit more stuff in it. Let’s go to another document. Here you can actually see one with international characters so if you click on that, you’ll see we actually maintain international characters across this. Let’s pick on maybe an actual report like a Word document.
Here you’ll see a Word document, I click on this, it’ll actually take me to the physical Word document, I can download it, open it, work with it just like I did before. That’s the essence of the documents, files, binary files, photos, things like that inside of … From Jive. If I click on this, you’ll see an actual photo is uploaded. Discussions are a little different. They’ll go to a separate library, it’s actually a discussion library in SharePoint so we have native discussions and if I pick one of these, you can see it has an image inside of it. Here’s a picture of a ring, obviously, and some commentary around what that was. It’s an actual discussion, a thread of discussions all represented here in native SharePoint form. You can see the 4 replies.
If I go, let’s pick a different one. Here’s also some discussion that has some of the representation of some of the international characters. Again, a lot of this was tested content so it really used just to make sure all that was working properly. Let’s go into the blog site. We’ll actually transition into the blog site, which is a separate template. If you scroll down, you’ll see the order that they were actually created in. You’ll see attachments. If I click on one of these, you’ll actually be able to open and interact with the attachment. See that downloaded, see the images, more attachments. So on and so forth. All the blog content is maintained in a sequence and we actually do leverage the SharePoint blog because why not use standard SharePoint functionality to do what it’s designed to do?
The next real step from here would be to turn things loose to validation so we would go into a deeper look at the counts to see did all the counts of content from Jive, does it match up with what we have in SharePoint, and then we do a much deeper dive in terms of looking at the links. We actually have another utility we use that validates all the links, all the images, making sure things got migrated as they were expected to be and then if there are any issues at all, we have a way to repair that stuff but typically we like to try and validate and get it right upfront.
That’s basically the essence of the migration and hopefully, this makes sense to you. I’m sure you guys may have some questions but at least you get a sense of how this all fits together and works so thanks for watching.
One thing I forgot to mention earlier or forgot to tie back to is that we looked at a particular discussion in Jive and wanted to see what that looked like in Jive. If we go back to SharePoint, you see Lisa’s awesome discussion. The same basic structure, everything maintained. Again, not really real content here. It’s just more of a test content just to show that we can move stuff around really for test cases but just wanted to kind of bring closure to that, make sure you guys see the resulting content.
