ludicrous-mode.jpg

Ludicrous Mode – Update on the Jive to SharePoint Migration Tool

Danny:Hi. This is Danny Ryan and welcome to the ThreeWill Podcast. I have procured Chris Edwards for your listening enjoyment. He has been quite busy lately.

 

Chris:Yes, very much so.

 

Danny:He’s been very busy, so I really appreciate you taking the time to do this. I’ve been talking to a lot of people about the Jive, the SharePoint migration tool. I’m anxious to get an update on what are the things you’ve been doing recently, maybe hit it at a high level and let’s dive in to some particular things. I know you’ve added some additional features to it. What’s been going on lately?

 

Chris:Just a lot of activity on the Jive, the SharePoint migration tool. Lots of things more so around just refining the process. Not so much the tool. Actually, I’ve got quite a bit of updates on the tool but more of the process of how we go about capturing inventory, really refining the inventory in such a way that people can understand what is out there in Jive. What do people have that they really care about, and how do they make decisions whether or not a particular Jive place or a collection of Jive content needs to stay or go. We’ve done some improvements really with the process to help make some of those decisions as well as understand when we do move a Jive place, how do we know if we did it correctly?

 

It’s great to be able to move content, but when we move content, what if only fifty percent of it came across? How do you know that? How do you know if eighty percent of it came across? If only eighty percent of it comes across, is that a big deal or not? Do we need to care about those exceptions and care about things that happen? A lot of more improvements around the tool, really around understanding the process and capturing ins and outs and making sure people are able to make really wise decisions about the content itself.

 

Danny:It sounds like you have people you take a look at, you have a way of understanding what’s currently in their Jive environment so that they can sort of … Because part of this process may be pruning some of it or not moving content over?

 

Chris:Yeah. One of the things that people may want to do is look at when was the last time a place or a particular piece of content within a place … When I say place, I mean this is a Jive terminology for any kind of container in Jive. One of the things we were able to expose now is when was something last modified in the context of a place? A lot of companies, especially the bigger companies, they want to have a retention plan in play. They don’t want to move stuff over that was last touched more than two years ago. We can now identify that a little bit better than we could before and really, it maybe that they want to archive it. The way the tool has been designed, I mean we talked about this last time, is that the tool still maintains the same design where we pull content out of Jive and we store it into either database and/or a file system. That’s kind of the archive, if you will. That’s still in play.

 

Whether or not we move it forward though, that same content, we transform it into something SharePoint can understand and push it into SharePoint which is our upload piece, we actually have terms for these things now, then … It’s crazy who that kind of stuff works. We can actually make decisions on whether or not something goes to SharePoint or not, and better decisions, not just everything. On top of the process and just being able to measure what are we doing well, what do we need to triage as an exception, we’re getting timeouts all over the place, how do we go about fixing that? Then, minimizing how much redo we have to do. We don’t want to have to redo everything that we find an error later on in the process. We want to minimize that. We’re not billing for that time. We want to make sure we’re using our customer’s money wisely and really making sure that we can wise decisions, enabling them, and then also making sure that they get their content in SharePoint and they’re happy with what they get. That’s all part of that.

 

We really have made a lot of improvements to the tool with regards to performance and pulling content out of Jive. We got more respect around the tool now. Before, it was kind of a loosey-goosey, here are the command lines, very … I thought it was good, I just threw it out there. It was quickly put together. Now we actually sat back and said, “Okay, what do we really need to be pulling out of Jive? How do we need to control it? Do we need to pull everything? Do we need to pull it by type? Do we need to be able to pull things by maybe something that was last modified by a certain time frame? We can specify these different parameters and really hone in on what we want to pull out of Jive.

 

We also have added multithreading to the tool. Now, the tool doesn’t just go after one place in Jive at a time. It can actually go after a whole collection of them based on how many processes around whatever server or whatever environment you’re running on, it can actually utilize those processors. Instead of being restricted by the latency of Jive and just pulling one place at a time, which is extremely slow, we can actually queue up and run multiple pulls from Jive simultaneously.

 

Danny:In the command line, do you like /ludicrous mode?

 

Chris:Exactly, ludi mode.

 

Danny:Ludi mode.

 

Chris:I put it in ludi mode and it just pulls it all down in two seconds.

 

Danny:Then that server starts warming up and it starts …

 

Chris:Yeah.

 

Danny:Nice.

 

Chris:You can really see it rip and do stuff. It’s trying to get a balance of what can we really pull out of Jive? How fast can we pull stuff down without causing it to break as well? It’s kind of getting that, trying to get to that balance point.

 

Danny:When we’re moving over, so we’re moving over like a batch of … We’re probably working with a client to decide what’s coming over first, is that correct?

 

Chris:That’s correct.

 

Danny:Then, once that moves over, there’s some … I’m sure there’s some remediation on our side and then on their side, some … Are they going through in some acceptance? “Yes, my content has been moved over properly,” is that what goes on/

 

Chris:Yes. Typically the customers’ responsible for validating their own content, but we put a lot of stuff and try to put a lot of stuff in place and really structure the roles for the … Like I said, I’m more of the migration engineer and tool, I make changes to the tool but we actually have a migration engineer person now that actually is just running the migrations. That’s all that their role is and they’re managing the exceptions and monitoring things before the customer even sees the content. They can help direct, “Oh, we have a lot of timeouts. Maybe something was going on in Jive. Maybe things were happening at that time. We need to go back through and check to see if any code needs to be modified to fix this.”

 

Really identifying patterns and what kind of issues are out there, so we can decide if it has to be redone or not, before even the customer gets it. Then the customer would get it, we would inform them that we’re going to get a dashboard format into our regular format where they could say, “These are the new places that are available in SharePoint.” We’ll have a look. If we find issues, then we’ll track it and the migration engineer will basically go to work with individuals to do that, and see do we need to redo things or not? We got more of process. We have to because we’re dealing with a lot more volume of content now. When we originally did the migration, it was for ourselves. We don’t have so much content. If we had to do manual steps, manual process, we did it.

 

Danny:How many years ago did we do that?

 

Chris:Two or three years ago.

 

Danny:Two or three years ago?

 

Chris:Yeah. It was the first kind of iteration of the tool itself. Since then, the tool has basically stayed in the same overall form. It’s still a console-based tool. It was built that way on purpose to keep it simple.

 

Danny:It’s just like JavaScript console apps are coming back.

 

Chris:Exactly. There’s no reason that … I mean, why bring an interface into this if it’s not needed. We have found the need for an interface in certain ways like when people want to manage what places go into Jive and we’re dealing with a spreadsheet right now.

 

Danny:That’s great. People are very familiar with working with a spreadsheet as well.

 

Chris:It’s great but it’s unwieldy as the same time. There’ll eventually be some improvements there where we can not be so tied to the spreadsheet and make it a little bit more … At least that’s my hope. Make it a little bit more of a user interface on that part of it. Can never guarantee that particular functionality is going to happen. It really depends on how many people are interested in the tool, how often we’ll use it and whether or not we really see the bang for doing that.

 

Danny:When you’re doing this, you must pull all of the user information over first before the content? Is that correct? What’s the sort of high level series of what happens?

 

Chris:Typically, we focus on inventory first. We talked about this a little bit the last …

 

Danny:You can say something twice. I will forget. Actually if somebody listen to it, you don’t have to hit it too hard, but go ahead.

 

Chris:That’s cool. Typically, we pull, we do a getPeople. We’d pull the people in Jive first because everything is dependent on the person. Even the creation of a place, even the specification of a place. When I say place, in Jive, it’s the concept of blogs, spaces, groups, and projects. Those are the kind of four container text. Place is just a generic term for that, so I keep saying that every time. Essentially we pull the people, we pull the places, and then we pull what’s called the metadata run of content. We pull the high level of content into our database so we can get content counts. We know kind of at breadth what’s out there, what was last modified at the content level and at a place level, and we produce a spreadsheet out of that. We take that and we give that to the customer so they can say, “Okay, these are the particular places that really have been modified most recently, which ones have not been touched forever.”

 

They have a lot of information as well as counts as well as the size of each of the places, I mean, all that kind of stuff’s in the spreadsheet. They can take that and make their decisions. Then we batch it like you said earlier into logical chunks. We found that a logical chunk of like fifty places at a time is a nice focused batch. We may ramp that up as we really get further into this and work with larger customers. A batch of fifty at a time seems to be pretty reasonable because you can report updates on those, you can monitor exceptions on those, and it’s a consumable amount to say, “Here customer, here’s another batch of fifty. You can communicate this to your end users.” It seems to be a pretty nice manageable amount. That’s pretty much how it works.

 

Danny:It sounds like here’s a couple of new folks that are working with you. Is that like Lisa and Brendan?

 

Chris:Mainly Lisa. I have Gary working with us a little bit as well, doing more of some performance improvements and helping with some of that.

 

Danny:What about, are there different content types? Have there been some content types nuance that we’re supporting? I know we’re giving some estimates around where a blog post maybe coming.

 

Chris:When we originally wrote the tool, it was focused on three different containers. I kind of rattled them off earlier. There was the blog, project, group, and space. Originally blog was not part of our mix. It wasn’t really treated as a primary content type inside of Jive. We were really focusing on the spaces, projects, and groups. That really came over when we originated the tool for ourselves. I don’t think blog as a container type was really even around that time. We’ve been improving the tool. We got some updates to it that actually include blogs as a primary container type now as well and bringing over that content as well too. That’s some improvements to the tool and really making sure we can cover any type of place that’s in Jive. We have a story to deal with that. We focused on tasks. We’ve pulled discussions. We’ve actually changed the tool now so instead of generating pages like Wiki pages, things like that for discussions, it actually generates SharePoint discussions and real thread SharePoint discussions in SharePoint.

 

Danny:Nice.

 

Chris:It’s nice because it actually can be begin to introduce as a discussion. A discussion in Jive can be moved over to SharePoint and continued to be used as a discussion in SharePoint. The user information’s maintained, the timestamp’s maintained and the whole threaded nature of the discussion is maintained. That’s definitely the super improvement to the tool.

 

Danny:I used discussions. I used one like, I think the last time was six years ago. I mean, it was [inaudible 00:12:42]. I don’t remember.

 

Chris:Yeah. It’s not …

 

Danny:I don’t remember the la- … I’m being just upfront. It seems like that’s been something that again with the overlap between SharePoint and Yammer, more recently just got lot of the discussions go on inside of Yammer.

 

Chris:Right.

 

Danny:At least this gives you an archive or the content and you can pick up the Yammer discussion off of it as well.

 

Chris:You can do that.

 

Danny:You can refer to it or, you know, stuff.

 

Chris:You could use it by its pure nature, you could use it as a discussion in SharePoint, so that doesn’t really end that, but yeah you said it, it’s a great launching point to start from Yammer to say, “Okay, I want to point to this discussion and continue from there.” It works either way. It maintains the integrity too of the threaded nature of it and kind of how things were created in Jive. It matches a little bit better. Before, we were doing things where we would create these pages and it was more of a static pull. You would have to go to something like a Yammer. It was really your only option that can continue the conversation. Now we have a little bit more of a rich set of options to go continue the conversation.

 

Danny:Jive has the concept of secret groups. We’re handling those as well?

 

Chris:We’re actually capturing permissions. In terms of when I say permissions, really we’re capturing the users at the place level. How many members are in different Jive places? We’re actually able to capture that detail now in our database. That’s another piece of archive information we have. We could also play that forward. If something was a secret or private group in Jive, we can actually provision the site in SharePoint, apply the actual permissions in terms of who is a member of what in SharePoint, and then pull in the content. That way, we’re not actually opening up content to folks that shouldn’t see it. We’re actually able to leverage and move over the people and the people membership references from Jive into SharePoint.

 

Danny:Then those people would, you’d have to have the acceptance on their side, you’d have to have somebody who has access to the secret group to say, “Yes, the content’s moved over.” You have a secret group, you moved it over, how do you know …

 

Chris:Secret private groups in terms of SharePoint, there’s really no kind of differences between those. They’re basically the same thing.

 

Danny:Once it’s moved over to validate … Let’s say there’s a super secret HR discussion group, three members in that group that are over in Jive, it gets pushed over into SharePoint, then you have to have one of those three people to go and verify that the content did get moved over properly.

 

Chris:Right. As part of that spreadsheet I mentioned earlier, we actually have a list of all those types of users. For those highly sensitive places, it’s still up to the customer to communicate to their end users when it’s time to go look at it. We don’t add users to a site in SharePoint unless they were part of the appropriate place in Jive.

 

Danny:You’ve added, done some tweaks with multithreading.

 

Chris:Yes. [crosstalk].

 

Danny:Some of the [crosstalk]. What are we at a high level, just at a high level, what does that mean? Like you’re moving over a certain number of sites per hour or what is that?

 

Chris:This is for like the content metadata run like I talked about earlier where we just wanted not necessarily to pull down all binaries but we just iterate through all the content in Jive. We’ve been able to do something that previously took like a day, day and a half, I’ve seen it go down to like an hour, hour and a half. It’s a substantial difference.

 

Danny:Nice. Then the file content, it’s dependent upon how quickly the bits go over the … Is it pretty quick?

 

Chris:Pretty quick as well. Yeah. Typically we’re pulling down content and all the binaries that come with all that. Pretty much, everything that, let’s say from a batch of fifty like I talked about earlier.

 

Danny:Sure.

 

Chris:A batch of fifty places, let’s say they’re not super large places but let’s say a batch of fifty, we could pull that down in say like fifteen minutes. Before, that might have taken us half a day to do that just because we’re doing it at a very single-threaded way. There’s just a lot of latency there that we’re just not taking advantage of the machine that was running the tool. Really try to take full advantage of the machine and what Jive could actually accept for throughput. That way, we’re able to really trim that down. It’s important because people that are wanting to come off Jive, they want to get off there within a timely manner. It can be a major risk if they’re running up against the licensing end date and they’ve got a lot of content. We want to be able to get that content down quickly and be able to, I don’t know what we’re going to do with it.

 

Danny:Is there any plans to, because I know with Jive you can request a backup, let’s say when you’re moving off of Jive. Are there any plans for us to, instead of working with our API, to work directly with the backup to do this?

 

Chris:I’ve done that myself. Just poking into the database, but we don’t have any plans at the moment that allows us to programmatically look at that.

 

Danny:It might be interesting for us to look at that because if that’s … Because APIs change, right?

 

Chris:Sure.

 

Danny:Just in case like as a backup plan, if you can get the database backup of the whole thing and …

 

Chris:Can we take in a full content out there and transform it the same way?

 

Danny:Can we take in full content and then … Yeah.

 

Chris:That’s one of the nice things. The way we’ve designed this is that if you could read that database programmatically then you could basically push it through the same procedures we currently have for the transform piece and the uploads. All that would stay the same.

 

Danny:The only reason why I said it is it’s also a backup plan in case they’re not able to get out in time and we’re not able to get everything out if we can work with the database backup. That gives an alternative planning for people.

 

Chris:Exactly.

 

Danny:Just a thought I tried to come up with.

 

Chris:That’s good.

 

Danny:Such a little thing.

 

Chris:Backlog.

 

Danny:That’s probably the worst type ever. You would say that to me like, “Danny, that is the worst idea.” You’re going to leave the room and go tell people, “Danny said we’re going to [crosstalk].”

 

Chris:I think [crosstalk]. No.

 

Danny:He obviously hasn’t touched Visual Studio in over seven years.

 

Chris:It’s all good.

 

Danny:It’s all good. Thank you.

 

Chris:You know what, it’s actually a good idea. If companies are coming up with that, running into that risk of that restriction, we want to have as many options. It’s really one of those things if people really need it, we’ll do it. Probably not going to be something we’re going to focus on right now.

 

Danny:You’re now moving over to doing some getting started workshops, another some that we’re throwing on your calendar to help out. I think there’s some other folks who are jumping into the fray as well …

 

Chris:Exactly.

 

Danny:… to help out. That’s going to be good for you also just to see a couple, you know, more environments, see what people are doing and making the tool even better.

 

Chris:Making the tool better, getting other folks familiar with the tool, and really getting their thoughts too on ways that it could improve. Like I said, it was designed quickly. It was put together quickly to accomplish a task. We definitely refined it and made it a whole lot better, but it doesn’t mean it couldn’t be even better.

 

Danny:A lot of products come from scratch in your back. It’s amazing.

 

Chris:It’s solving a problem, right?

 

Danny:Yeah, solving a problem.

 

Chris:Yeah.

 

Danny:Anything else before I let you go and don’t see you for another quarter? Well, I’ll see you I guess at the kick-offs for these things.

 

Chris:Sure. No, I mean, that’s the basic gist. We are putting together some documentation to better define some of the stuff we’re talking about so it’s not just me rattling this stuff off on a recording like this. We’re actually going to put together some book and some details, so if someone wanted to pick this up, they could understand it in much more detail.

 

Danny:I get request all the time from folks in foreign lands asking for this as a product and I know right now it’s still a fluid thing, but maybe even- … I’m not saying we would never package it up as a product, but for right now, I think it’s best we’re doing the services along with it and then including the tool itself until we get it to a point where we’re happy with it.

 

Chris:Yeah. We talked a little bit about being able to monitor exceptions, monitor the … I mean, I just don’t think it’s at a state where I’d feel comfortable with handing this off to someone and they go run it. I want to be pretty confident that they’re going to be successful with the tool. It definitely requires some hand-holding. It doesn’t mean one day, we’ll just have to wait and see what it looks like and what the demand is.

 

Danny:Absolutely.

 

Chris:If it’s really going to help someone, we’ll make that decision then.

 

Danny:Awesome. I hope everybody enjoyed this especially if you’re looking at moving from Jive to SharePoint. Just a little update. Probably the next time, I’ll probably just do another update on this. This really helps me because when people are contacting us, unfortunately I’m the first line of defense.

 

Chris:I don’t know what.

 

Danny:I don’t know what it does actually but at least if we get an update once a quarter, I kind of can say what it does. This really helps me out, so thank you for taking the time to do this.

 

Chris:Anytime. Thanks.

 

Danny:Awesome. We have recently added some intro and outro music to our podcast.

 

Chris:Nice.

 

Danny:We’re getting there. I’m slowly putting a spit shine on it. It’s getting there. Thanks everybody for listening. Please drop by our threewill.com site. You’ll see a whole section on migration. Come drop by and let me know what you’re trying to do, what you’re trying to migrate from, where you’re trying to go. Really appreciate everybody reaching out there and have some real good leads come to the website. We really appreciate you taking the time to reach out. Thank you so much for listening to this podcast and have a super day. Bye.

 


Related Posts

Chris EdwardsLudicrous Mode – Update on the Jive to SharePoint Migration Tool

1 comment

Join the conversation

Join the conversation

This site uses Akismet to reduce spam. Learn how your comment data is processed.