Wondered how to schedule data import from a REST webservice?

Discover an efficient method for synchronizing data from a service on a recurring basis. Learn how to achieve this seamlessly and effortlessly as I provide step-by-step instructions on a streamlined approach.

The Use Case

As always, you need a use case to start with, why else would you build something innit. Here goes…let’s say you want to port incident data from instance 2 every Saturday morning 5 AM (why 5 AM you ask?! cause that’s when 98% of the people are sleeping you dummy! :D). Instance 2 has provided a couple of web-services that instance 1 can consume. You need to hit these APIs, retrieve the data, massage it and store the data in instance 1’s incident table….Easy peasy!

Creating the import set table

This my friend is a crucial step which has a tiny little nuance tied to it. What you’re trying to achieve here is to create a table that is recognized by modules where a dataset table is referenced (transform maps, load data, etc).

My goal is to read the Number, Short Description and the Description fields from the data I receive from instance 2 and then map that to the incident records that I create in instance 1. For this purpose, I have introduced 3 new fields Number, Short Description and Description in my import set table.

Let’s get a scheduler going

I’ve created a scheduled job called ‘Import Incident Data’ which invokes a Script Include that contains the logic that calls the web service. This guy runs every Saturday morning (like I said earlier).

Now the Inc_Import script include acts like the data transfer object and looks like this,

var Inc_Import = Class.create();
Inc_Import.prototype = {
    initialize: function() {
    },
	importIncidents: function() {
		var irm = new Incident_RestMessage().getIncidents();
		if(irm != null) {
			if(irm.getStatusCode() == 200) {
				var data = JSON.parse(irm.getBody());
				data.forEach(function(incident,index) {
					var gr = new GlideRecord('u_inc_data_import_set');
					gr.u_number = incident.number;
					gr.u_short_description = incident.short_description;
					gr.u_description = incident.description;
					gr.insert();
				});	
			}
		}
	},
    type: 'Inc_Import'
};

The Incident_RestMessage script include responsible for making the call to the web service looks like this,

var Incident_RestMessage = Class.create();
Incident_RestMessage.prototype = {
    initialize: function() {
    },
	getIncidents: function() {
		try { 
			var r = new sn_ws.RESTMessageV2('Incident_RestMessage', 'GET');
			
			var response = r.execute();
			var responseBody = response.getBody();
			Incident_Logger.get().debug("Reponse Body Inside Get Incidents is " + responseBody);
			
			var httpStatus = response.getStatusCode();
			Incident_Logger.get().debug("Response Status Inside Get Incidents is " + httpStatus);
			
			var httpsErrorMessage = response.getErrorMessage();
			Incident_Logger.get().debug("ErrorMessage " + httpsErrorMessage);
		}
		catch(ex) {
			var message = ex.message;
			Incident_Logger.get().debug("ErrorMessage " + message);
		}
		
		return response;
	},
    type: 'Incident_RestMessage'
};

I also have this logger script include that logs errors/messages incase something goes wayward,

var Incident_Logger = Class.create();
Incident_Logger.prototype = {
    initialize: function() {
    },
	  debug: function(msg){ this._getLogObj().logDebug(msg); },
	  err: function(msg){ this._getLogObj().logErr(msg); },
	  info: function(msg){ this._getLogObj().logInfo(msg); },
	  _getLogObj: function(){
		var logger = new GSLog("com.thesnowmonk.incident.log_level", "LINKEDIN_INC"); return logger;
	  },
    type: 'Incident_Logger'
};
Incident_Logger.get = function(){ return new Incident_Logger();};

Last, but not the least…the transform map

Now that we have our data added to our import set table, it’s time to transform the data and map it to the actual incident table. What better way to do it, than a transform map.

That’s all folks! give it a rip! Next time the schedule job runs, it picks up data from the web service and maps it to instance 1’s incident table…Automagically!