Danny serves as Vice President of Marketing at ThreeWill. His primary responsibilities are to make sure that we are building partnerships with the right clients and getting out the message about how we can help clients.
Danny Ryan: Hi this is Danny Ryan and welcome to the ThreeWill Podcast. Today I’ve got Kirk Liemohn here with me. Kirk is a Principal Software Engineer here at ThreeWill. Thanks for joining me Kirk.
Kirk Liemohn: Hello. Good to be here.
Danny Ryan: Great. I’m pulling you in for … We’re going to hit a technical subject today and that subject is web jobs. These are Azure Web Jobs. Is that correct?
Kirk Liemohn: That is correct.
Danny Ryan: Awesome. I’d love to learn a little bit more about what this is. Maybe a little bit of history of how you solved this problem before in the past. Some of the benefits of the way you’re taking your approach today. Tell me a little, maybe just starting at a high level, what is an Azure Web Job?
Kirk Liemohn: It’s just a task that runs in the background on Azure, within an Azure app. You can use it just to run anything that might take a long amount of time maybe, or just something that you want to run on a periodic basis.
Danny Ryan: Great. What would … is there an equivalent type of thing on SharePoint as far as the way you would approach this?
Kirk Liemohn: Yeah. In the past, in SharePoint we would do a timer job, definitely. If we had access to do that, you can’t do that in SharePoint online. You could write your own timer job in SharePoint. You could do a scheduled task on a Windows server as well or even a client machine if you wanted to. Those are ways to in the past and I’ve got a Unix background from back in the ’90s and back then it would have been a ChronJob.
Danny Ryan: Excellent. Sort of the history of the way that you periodically ran a piece of software or …
Kirk Liemohn: Yep.
Danny Ryan: I know when we we’re talking about prepping for this you also mentioned that you could use this for long running processes as well?
Kirk Liemohn: Yeah. Say if you’ve got a website, especially if it’s running Azure and you’re using the PaaS solution for doing websites in Azure. It needs to do some processing and for whatever reason that processing takes more time than you want to make the user wait. Even if it’s just a few seconds, might be enough to say, “Hey, let’s do this asynchronously.” You could push it off to a web job. If it certainly might take minutes or hours you would want to consider a web job for background processing.
It may not be something you do on a scheduled basis. It may be initiated by a user. If you we’re to, on that website, try to spawn a thread so that the user didn’t have to wait that delay. IIS, if you’re using IIS, it’ll potentially kill your thread because it thinks, “Well I’ve already returned to the user and I don’t need to be having anything else going on. I see there’s something else going on but I don’t care. I’ve already returned control to the user.” It has no use for your thread you’ve created so it’s better to do it with something like a background process via web job.
Danny Ryan: Tell me a little bit about some of the things that you learned while doing this. Talk a little bit about how did you package this up and get it on Azure.
Kirk Liemohn: First off, this is my first time doing a web job. I’ve wanted to do them in the past. I’ve had some times I didn’t have to do them, but I kind of wanted to do them. This is a time I really needed to. I am doing a migration to Microsoft 365 and we’re concerned about throttling. This is SharePoint, this is a SharePoint migration. We know that we might get throttled by SharePoint online. We want to know when we’re being throttled.
The best way we can do that is to look at the logs for the migration tool and read that information and find out if there’s an entry in the log that indicates that we’re being throttled. I’ve got several migrations running at once on several servers. I didn’t want them to each go through and look for this because it is a little costly. I have to do a full text index of the content and a query across that. I’ve decided that I want to run it on a periodic basis. A job that will do that check and then it will post something to a simpler database that says, “Yep, I’ve been throttled or not.”
It’ll be a lot easier to say, create a dashboard that shows what the throttle status is, if there’s been any throttles recently, as well as the migration … The servers that are doing migrations they can do that check and say, “Oh, if we’ve been throttled recently I don’t want to start another migration job. I’m going to hold off until I’ve got no throttles within the last hour or something.”
Danny Ryan: Awesome. How are you alerted about this? You’ve mentioned a dashboard. I guess you could do it through a dashboard, but how else?
Kirk Liemohn: I haven’t created the dashboard yet, I might do that. I use Visual Studio, is what put it into the Azure Web Job for me. I started out making it as a PowerShell script and my PowerShell script relied on a commandlet that for Sequel that I had to use a couple of, actually three, MSIs to get that working. I didn’t know how to install MSI’s in a web job. I don’t think you can. I know you can upload a bunch of files and it was simpler to me to just switch it over to a console app.
I switched to a console app; this console app was written in Visual Studio. Visual Studio was real easy to just right click on the project and say, “Publish this web job.” There’s a few settings you have there, how often you want it to run. Then from Azure, the Azure console, admin console, you can see the status like any output from your console as well … Your console app.
Danny Ryan: You mentioned that this runs every hour, but that’s primarily because of the type of Azure account that you have. Is that correct?
Kirk Liemohn: That’s right. This will run, I have it running once an hour. That’s the fastest I can have it run since I think we’re on the basic, I think it’s the basic plan for our Azure web app. If I we’re to change that to standard or premium I could do it more often. We may do it for that reason, we haven’t decided that we need to go to that level yet. That is correct. While it’s running it simply does the query, the full text query. It’s against SQL which is running on Azure machines so I have to have the access to that sequel server over the internet so I’ve got special accounts and ports for that. Then if it finds something it will log something to a different database table and it will also send an email. I use SendGrid for that. Which I’ve used SendGrid in the past with Azure so it was pretty easy to set that up.
Danny Ryan: It’s a pretty easy configuration for that?
Kirk Liemohn: Yeah, it probably took me 30 minutes and I hadn’t used SendGrid in several months, maybe a year. It wasn’t fresh in my memory but about 30 minutes to figure out how to send email, basically. Set it up and configure it and all that.
Danny Ryan: Its funny you mention this is the first one you’ve done. I think that’s the best time to give advice to other people who might be doing this for the first time. Any other pointers that you would give to people for writing Azure web apps, web jobs?
Kirk Liemohn: Yeah, web jobs. The first thing is try and understand what it relies on. If it’s relying on certain other pieces of software or something installed on your system then it may not be appropriate or you may need to find a way to get those up into Azure. If you need other files, in my case initially I needed some other commandlets for PowerShell and that wasn’t going to be the easiest path for me so just doing an all encompassing console app that knew how to talk to Send Grid and knew how to talk to databases was all I needed.
Danny Ryan: It’s funny it seems like console apps are coming back because I was talking to Chris yesterday and he wrote for the migration tool. The Jive to SharePoint migration tool. That’s as a console app as well. It seems like they’re coming back in fashion now.
Kirk Liemohn: Oh yeah. All of our stuff for migration is mostly PowerShell based and we run all that from consoles, so definitely. The gotcha is just try and understand what your program relies on. Try and understand what your needs are in terms of how often this thing needs to run. Those are kind of the two things that I ran up against.
Danny Ryan: That’s great. I appreciate you taking the time to do this. I know you’re busy on your project but hopefully this will help somebody who may be out there looking to write their first Azure Web Job. Thank you for doing this Kirk.
Kirk Liemohn: Sure. You’re welcome.
Danny Ryan: Thanks everybody for listening. Have a great day.