Oh my god. It's full of code!

Javascript

Create an Alexa skill in Node.Js and host it on Heroku

Ok, here we go.

So for the last couple weeks I’ve been totally enamored with working with Alexa. It’s really fun to have a programming project that can actually ‘interact’ with the real world. Until now I’ve just been using my hacked together web server with If This Then That commands to trigger events. After posting some of my work on Reddit, I got some encouragement to try and develop and actual Alexa skill, instead of having to piggyback off IFTTT. With some sample code, an awesome guy willing to help a bit along the way and more than a little caffeine I decided to give it a shot.

The Approach: Since I already have a decent handle on Node.Js and I was informed there are libraries for working with Alexa, I decided Node would be my language of choice for accomplishing this. I’ll be using alexa-app-server and alexa-app Node.Js libraries. I’ll be using 2 github repos, and hosting the end result on Heroku.

How it’s done: I’ll be honest, this is a reasonably complex project with about a million steps, so I’ll do my best to outline them all but forgive me if I gloss over a few things. We will be hosting our server in github, creating a repo for the server, making a sub module for the skill, and deploying it all to Heroku. Lets get started. First off, go and get yourself a github account and a Heroku account. If you haven’t used git yet, get that installed. Also install the heroku toolbelt (which may come with git, I can’t quite remember). Of course you’ll also need node.js and Node Package Manager (NPM) which odds are you already have.

If you don’t want to create all the code and such from scratch and want to just start with a functioning app, feel free to clone my test app https://github.com/Kenji776/AlexaServer1.git

Create a new directory for your project on your local machine for your project. Then head on over to github and create yourself a new project/repo. Call it AlexaServer or something similar. Go ahead and make it a public repo. Do not initialize it with a readme at this time. This is the repo where the core server code will live. It’s important to think of the server code as a separate component from each individual skill, they are distinct things. You should see this screen.

repo1

Open a command prompt and change to the directory you created for your project. Enter the commands show in the first section for creating a new repository. Once those commands are entered if you refresh the screen you shoud see your readme file there with the contents shown like this.

github1

Okay, now we are ready to get the Alexa-Server app, https://www.npmjs.com/package/alexa-app-server is what you are looking for. In your project directory type in

“npm install alexa-app-server”

This will take a few moments but should complete without any problems. In your project folder you’ll want to create a folder called apps. This is where each individual skill will live. We will cover that later. Now you’ll need to create your actual server file. Create a file called server.js. Put this in there.

'use strict';

var AlexaAppServer = require( 'alexa-app-server' );

var server = new AlexaAppServer( {
	httpsEnabled: false,
	port: process.env.PORT || 80
} );

server.start();

Pretty simple code overall. The weird bit of code in the port section is for Heroku (they give your app a port to use when hosted). If not on Heroku then it will default to using port 80. Now you need to create your Procfile. This is going to tell Heroku what to do when it tries to run your program. It should be a file named Procfile in the same directory with no file extension. The contents of which are simply

“web: node server.js”

without quotes. We will also want to create a package.json file. So again in your project directory run the command

npm init

This will run a script and it will ask you a few questions. Answer them and your package.json file should get generated. Go ahead and push this all into Github using the following command sequence.

git add .
git commit -m “added server.js and Procfile, along with alexa-app-server-dependency”
git push origin master

If you view your Github repo online you should see all your files there. It should look something like this.

first repo push

 

You can see all of our files got pushed in. Now with the server setup, it’s time to create our skill. Create another GitHub repo. Call this whatever you like, hopefully something descriptive of the skill you are making. In your command prompt get into the apps directory within your main project. In there create another folder with a name same or similar to your new GitHub repo. Follow the same steps as before to initialize the repo and do the initial commit/push. Now we are going to indicate to GitHub that the apps/test-skill folder is actually a sub module so any dependencies and such will be maintained within itself and not within the project root. To do this navigate to the root project folder and enter.

git submodule add https://github.com/Kenji776/AlexaTestSkill.git apps/test-skill

Replacing the github project url with the one for your skill, and the apps/test-skill with apps/whatever-your-skill-is-named. Now Git knows that this folder is a submodule, but NPM doesn’t know that. If you try and install anything using NPM for this skill it’s going to toss it into the root directory of the project. So we generate a package.json file for this skill and then NPM knows that this skill is a stand alone thing. So run

npm init

Again and go through all the questions again. This should generate a package.json file for your skill. Now we are ready to install the actual alexa-app package. So run…

npm install alexa-app –save

and you should see that the skill now has it’s own node_modules folder, in which is contained the alexa-app dependency. After this you’ll have to regenerate your package.json file again because you’ve now added a new dependency. Now it’s time to make our skill DO something. In your skill folder create a file called index.js. Just to get started as a ‘hello world’ kind of app, plug this into that file.

module.change_code = 1;
'use strict';

var alexa = require( 'alexa-app' );
var app = new alexa.app( 'test-skill' );


app.launch( function( request, response ) {
	response.say( 'Welcome to your test skill' ).reprompt( 'Way to go. You got it to run. Bad ass.' ).shouldEndSession( false );
} );


app.error = function( exception, request, response ) {
	console.log(exception)
	console.log(request);
	console.log(response);	
	response.say( 'Sorry an error occured ' + error.message);
};

app.intent('sayNumber',
  {
    "slots":{"number":"NUMBER"}
	,"utterances":[ 
		"say the number {1-100|number}",
		"give me the number {1-100|number}",
		"tell me the number {1-100|number}",
		"I want to hear you say the number {1-100|number}"]
  },
  function(request,response) {
    var number = request.slot('number');
    response.say("You asked for the number "+number);
  }
);

module.exports = app;

Now, if you aren’t familiar with how Alexa works, I’ll try and break it down for you real quick to explain what’s happening with this code. Every action a skill can perform is called an intent. It is called this because ideally there are many ways a person might trigger that function. The might say “say the number {number}” or they might say “give me the number {number}” or many other variations, but they all have the same intent. So hence the name. Your intent should account for most of the common ways a user a might try to invoke your functions. You do this by creating utterances. Each utterance represents something a user might say to make that function run. Slots are variables. You define the potential variables using slots, then use them in your utterances. There are different types of data a slot can contain, or you can create your own. Check out https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-kit-interaction-model-reference for more information on slots types. I’m still figuring them out myself a bit. So the intent is triggered by saying one of the utterances. The slot is populated with the number the person says, and after reading the variable from the request, it is read back to the user.

So now, believe it or not, your skill is usable. You can test it by heading by starting your server. Again in your command line shell within the root project directory type

node server.js

A console window should show up that looks like this.

skill running

Now in your browser you should be able to bring up an emulator/debugger page be heading to

http://yourserver/alexa/your-skill-name       (EX: http://localhost/alexa/test-skill)

You should get a page that looks like this.

emulator

Holy cow, we did it! We have a functioning skill. The only problem now is that there is no way for Alexa to ‘get to it’. As in, Alexa doesn’t know about this skill, so there is no way for echo to route requests to it (also, I find it mildly creepy that when referring to Alexa, I keep accidentally wanting to type ‘her’. Ugh). So now we have to put this skill somewhere were it’s accessible. That is where Heroku is going to come in. Sure we could host this on our local machine, setup port forwarding, handle the SSL certs ourselves, but who wants to have their skill always running on their local box? I’ll do a bit on how to do that later since it requires creating a self signed SSL certificate using openSSL and such but for now I’m going to focus on getting this sucker into a cloud host.

Oh by the way, once everything is working smoothly, you should commit and push it all into github. Remember, you’ll have to do a separate commit and push for your server, and for the skill submodule since they are officially different things as far as github is concerned, even though the skill is a sub directory of your server. Also, I’m still learning exactly how submodules work, but if for some reason it doesn’t seem like your submodule is updating properly you can navigate to the project root folder and try this series of commands.

git submodule foreach git pull origin master
git add .
git commit -m “updated submodule”
git push origin master

with some luck that should do it. You’ll know it’s all working right when you open up your server repo in github, click the apps folder and you see a greyish folder icon for the name of your skill. Clicking it should show you the skill with all the files in there. Like this.

submoduleNow, on to Heroku. This part is pretty easy comparatively. Head on over to Heroku and create a new app. As usual, name it whatever you like, preferably something descriptive. Next you’ll be in the deploy area. You’ll be able to specify where the code comes from. Since we are smart, and we used github you can actually just link this heroku project straight to your github project.

hreoku connect
Just like that! However, this automatic integration does not seem to include sub modules, so we will have to add our skill submodule from the command line. So again navigate to your project folder from the shell, and run this command to link your directory with the heroku project.

heroku git:remote -a alexatestapp

Of course replacing alexatestapp with whatever the name of your heroku app is. You should get back a response saying that the heroku remote has been added. Now we need to push it all into Heroku (which in retrospect may have made the above linking done from the website un-needed, but whatever. It doesn’t seem to hurt anything). So now run

git push heroku master

You should be treated with a whole bunch of console output about compressing, creating runtimes, resolving dependencies, blah blah blah. Now with some luck your app should work via your hosted Heroku app. Just like when hosted locally we can access the emulator/debugger by going to

https://alexatestapp.herokuapp.com/alexa/test-skill

Replacing the domain with your heroku app, and the skill with whatever your skill is named. If all has gone well it should operate just like it did on your local machine. We are now finally ready to let Amazon and Echo know about our new skill. So head on over to https://developer.amazon.com/edw/home.html#/skills/list (you’ll need to sign up for a developer account if you do not have one) and click the add new skill button.

app create 1

On the next page it’s going to ask you for your intent schema. This is just a JSON representation of your intents and their slots. Thankfully that handy debug page generates that for you. So it’s just copy paste the intent schema from the debug page into that box. It will ask you about custom slot types, and odds are if you are making a more complicated app you are going to want to make your own, but really all that is is creating a name for your data type, common values for it (no it doesn’t have to be every value) and saving it. Spend some time reading the docs on custom slot types if you need to. It’s also going to want sample utterances, which again that debug page generates for you, so again, just copy paste. You might get a few odd errors here about formatting, or wrong slot types. Just do your best to work through them.

model

 

Next up is setting up the HTTPS. Thankfully Heroku handles that for us, so we don’t have to worry about it! Just select

“My development endpoint is a subdomain of a domain that has a wildcard certificate from a certificate authority”

Next up us account linking. I haven’t figured that one out yet, still working on it, so for now we just have to say no. Next up is the test screen. If everything is gone well up to this point, it should work just peachy for ya.

testsay

Next you’ll be asked to input some meta data about your app. Description, keywords, logo, icon, etc. Just fill it out and hit next. Finally you’ll be asked about privacy info. For now you have just say no to everything, and come back to update that when you’ve got your million dollar app ready to go. Hit save and you should be just about ready to test through your Echo. If you open your Alexa app on your phone and look for your skill by name, you should find it and it should be automatically enabled. You should now be able to go to your echo, use the wake word, speak the invocation name and ask for a number. If all has gone well you’ll hear the wonderful sound of Alexa reading back your number to you. For example

“Alexa Ask Test Get My Number”

Alexa should respond with what you defined as the app start message in your skill. You can now say

“Say the number 10”

Which should then be repeated back to you. Congrats. You just created a cloud hosted, Node.Js Alexa Skill that is now ready for you to make do something awesome. Get out there and blow my mind. Good luck!


Making Amazon Echo/Alexa order me a pizza

UPDATE! I’ve started a github project for my server software. Still very alpha but you can check it out here: https://github.com/Kenji776/AlexaHomeHub

If you follow my blog, you might have caught my post yesterday about how I bought an Amazon Echo device, and have begun creating my own custom actions for it. I started small, simply making it call out to my web server by using If This Then That (IFTTT – a website that allows you to connect and integrate different services). I got it to connect to my door lock and unlock service I had written, and even got it to chromecast specific pre-setup videos to one of my TV’s using a command line tool. Feeling somewhat confident I decided it was time to take on something a little more in depth, but oh so worth it. I was going going to make Alexa order me a pizza.

If you are an Echo/Alexa user you might know that there is already support for ordering a pizza but only from Dominos. It uses some system of having a saved order, then tweeting a specific bit of info at the Dominos twitter account that is tied to your order which then places it. This has a few drawbacks. Primarily being that it orders you a Dominos pizza (sorry guys, in all fairness Dominos has gotten a lot better in recent years). Also it requires twitter integration, and as far as I know only supports one order that you have saved (I could be wrong). The using a saved order was a good idea as it streamlines and simplifies the ordering process quite nicely. I wanted to do something like this, but instead I wanted Sarpinos pizza, and I wanted to be able to pick from several different pre-created orders. Using my knowledge of browser automation that I picked up from my door lock/unlock project and my existing web server, I got to work.

First off, I had to figure out all the things that needed to happen. Off the bat I knew I’d be using their online ordering interface. They don’t have an API, so I knew I’d have to be automating browser interactions. Next I had to break down the process of ordering the pizza online step by step, all the HTML elements involved and how to interact with them. Then I would be able to automate those interactions using the Selenium library. So I went through the process like a normal person and created this list. At each step I inspected the HTML elements involved and recorded them so I could figure out how to identify them and interact with them later. I created an order and saved it as a favorite so next time I’d come back in I would be prompted if I’d like to order that again. From there I was able to create the following list of things I knew needed to get done.

Order Steps:

1) Invoke: https://order.gosarpinos.com/Login/

2) Wait For Load

3) On load populate credential fields:
	- <input class="text-box single-line valid" data-val="true" data-val-required="Email is required" id="Email" name="Email" type="email" value="" >
	- <input class="text-box single-line password valid" data-val="true" data-val-required="Password is required" id="Password" name="Password" type="password" value="" >

4) Click Login button
	- <input type="submit" value="Login" class="ui-button ui-widget ui-state-default ui-corner-all" role="button" aria-disabled="false">
	
5) Wait For Page Load

6) Find button with provided favourite id (428388)
	- <button class="wcFavoriteSelectFavoriteButton ui-button ui-widget ui-state-default ui-corner-all ui-button-text-only" data-favorite-id="428388" id="wiSelectFavorite428388" type="button" role="button" aria-disabled="false">

7) For For Delivery Popup to load
	- 

 

A fair amount of steps, but none of them super complicated. I knew I’d have to learn a bit more about Selenium as this interaction was definitely more complicated than the door lock code, and that one was already seemingly over complicated. Thankfully I did, and found out that in my previous attempt I had been mixing synchronous and async methods unknowingly hence leading to perceived complexity (I thought driver.wait() was a async method and you put everything that depended on it inside. Turns out it’s synchronous and once the condition inside is true the program continues. No wonder it was acting a little funny). I knew also since I was going to be passing in a fair amount of data (username, password, order id, credit card info, etc) that I should probably define an object which would have all the required properties, then just pass JSON into my web service that mirrored that object. This is what I came up with.

{
"action":"pizza",
"username":"xxxxxxxx@xxxx.com",
"password":"xxxxx",
"orderId":"428388",
"ccNumber":"xxxxxxxxxxxxxxxxxx",
"ccExpMonth":"03",
"ccExpYear":"2018",
"ccv":"xxxxx",
"tip":"5.00",
"ccZip":"xxxxx"
}

 

Obviously the sensitive values are blacked out, but you get the jist. My webserver is already primed to look for post requests that have a JSON payload. The ‘router’ code looks at the ‘action’ attribute to figure out what function to send the payload to. I created a new ‘pizza’ action and related function. Here is that function.

function orderPizza(orderObject,callback)
{
	var orderResult = new Object();
	
	//create instance of selenium web driver
	var driver = new webdriver.Builder().
	withCapabilities(webdriver.Capabilities.chrome()).
	build();

		
	//request the login page with the locks page as the return url
	driver.get('https://order.gosarpinos.com/Login');
	
	driver.findElement(webdriver.By.name('Email')).sendKeys(orderObject.username);
	driver.findElement(webdriver.By.name('Password')).sendKeys(orderObject.password);
	
	//find and click submit button
	driver.findElement(webdriver.By.css("input[type='submit']")).submit();

	console.log('Logged in as: ' + orderObject.username);

	//wait for order page to load
	driver.wait(function() {
		return driver.getTitle();
	},5000);
	
	console.log('Attempting To Choose Favorite Order With Id: ' + orderObject.orderId);
		
	//have to wait until the proper order button appears since it's in a dialog. If it isn't found after 5 seconds, fail. Otherwise click the corresponding order button
	//the favorite order button has an attribute 'type' of 'button' and a 'data-favorite-id' attribute with the id of that order
	
	driver.wait(function () {
		return driver.findElement(webdriver.By.css("div[aria-describedby='wiFavoriteListDialog']")).isDisplayed();
	}, 5000);
	
	
	driver.wait(function () {
		return driver.findElement(webdriver.By.css("button[data-favorite-id='"+orderObject.orderId+"']")).isDisplayed();
	}, 5000);
	
	driver.findElement(webdriver.By.css("button[data-favorite-id='"+orderObject.orderId+"']")).click();
			
	//after the button above is clicked, that dialog closes and another one opens. This one asks the user to select delivery or pickup. We want delivery
	//the delivery button has an attribute with a 'data-type' of 'WBD' and an attribute 'role' of 'button'
	driver.wait(function () {
		return driver.findElement(webdriver.By.css("button[data-type='WBD'][role='button']")).isDisplayed();
	}, 5000);
	
	driver.findElement(webdriver.By.css("button[data-type='WBD'][role='button']")).click();
	
	//ensure the checkout button is visible
	driver.wait(function () {
		return driver.findElement(webdriver.By.css("button[id='wiLayoutColumnGuestcheckCheckoutBottom'][role='button']")).isDisplayed();
	}, 5000);
	
	//hacky method to ensure that the modal dialog should now be gone and we can click the checkout button
	driver.sleep(2000);
	
	driver.findElement(webdriver.By.css("button[id='wiLayoutColumnGuestcheckCheckoutBottom'][role='button']")).click(); 
	
	//then the browser will move to the order screen. Once it loads we have to populate the order field data

	//wait for payment page to load
	driver.wait(function() {
		return driver.getTitle();
	},5000);	

	//wait until the pay by credit card button shows up.
	driver.wait(function () {
		return driver.findElement(webdriver.By.id("wiCheckoutPaymentCreditCard")).isDisplayed();
	}, 5000);

	//check the pay by credit card radio button
	driver.findElement(webdriver.By.id("wiCheckoutPaymentCreditCard")).click(); 
	
	//wait until the credit card number box shows up
	driver.wait(function () {
		return driver.findElement(webdriver.By.id("Payment_CCNumber")).isDisplayed();
	}, 5000);
	
	//populate the form fields
	driver.findElement(webdriver.By.name('Payment_CCNumber')).sendKeys(orderObject.ccNumber);

	//stupid jQuery ui selects are impossible to set with normal selenium since the original select is hidden. So use an execute script to set em.
	driver.executeScript("$('#Payment_ExpMonth').val("+orderObject.ccExpMonth+");");

	driver.executeScript("$('#Payment_ExpYear').val("+orderObject.ccExpYear+");");
		
	driver.findElement(webdriver.By.name('Payment_CVV')).sendKeys(orderObject.ccv);
	//driver.findElement(webdriver.By.name('Payment_CCTip')).sendKeys(parseFloat(orderObject.tip));
	driver.findElement(webdriver.By.name('Payment_AVSZip')).sendKeys(orderObject.ccZip);
	
	driver.executeScript("$('#Payment_CCTip').val("+parseFloat(orderObject.tip)+");");
	
	driver.findElement(webdriver.By.id('wiCheckoutButtonNext')).click();

	//wait for confirmation page to load.
	driver.wait(function() {
		return driver.getTitle();
	},5000);

	
	driver.findElement(webdriver.By.id("wiPlaceOrderNow")).click();	

	driver.wait(function() {
		return driver.getTitle();
	},5000).then(function(){
		console.log('Ordering complete!');
		orderResult.success = true;
		orderResult.message = 'Order Placed Successfully';
		callback(orderResult);			
	});
	

}

 

Now if that code seems a little dense or confusing, don’t feel bad. It took me several hours of trial and error to figure it out, especially then it came to setting the select list values, and getting the script to wait while various elements where created and destroyed by the page. Selenium has this super fun behavior where if you ever try and reference an element that doesn’t exist, the whole script goes down in flames. In response to that I made my code very ‘defensive’ checking to make sure elements that are required frequently before attempting to interact with them.

With the script created and integrated into my web server ‘router’ I was ready to get IFTTT to invoke it. Once again it was as simple as creating a new recipe with Alexa as the If and the Maker make a web request feature as the do.

pizza 1pizza2

You can see that with the combination of specific phrases and the fact that you can have multiple saved orders, it would be easy to setup many different possibilities. My roommate is even going to create his own IFTTT account and link it to my Alexa. Then he can create his own orders, specify his own credit card information in the JSON payload, and order whenever he wants using the same device but have his own information. The next step I think is to encrypt the JSON payloads which contain the credit card info and then decrypt them when the arrive at my server. That way I’m not storing my CC info in plain text anywhere which right now is a bit of a concern. This was mostly just proof of concept stuff last night, but I was too excited not to share this as soon as I could, so some of the ‘polish’ features are missing but overall I think it’s a damn good start. Now if you’ll excuse me, I’m going to get myself a pizza.

Update: Adding encryption was pretty easy. First I got some encrypt and decrypt functions set up. Like this.
.

// Nodejs encryption with CTR
var crypto = require('crypto'),
    algorithm = 'aes-256-ctr',
    password = 'xxxxxxxxxxxxxxxxxxxxxxxxxx';

function encrypt(text){
  var cipher = crypto.createCipher(algorithm,password)
  var crypted = cipher.update(text,'utf8','hex')
  crypted += cipher.final('hex');
  return crypted;
}
 
function decrypt(text){
  var decipher = crypto.createDecipher(algorithm,password)
  var dec = decipher.update(text,'hex','utf8')
  dec += decipher.final('utf8');
  return dec;
}

Then update my incoming data object so that the encrypted data was in its own property so I could still tell what kind of request it was without having to decrypt the payload first (since my webserver supports other, unencrypted calls).

"action":"pizza",
"data":"f613f8f479bad299bdfedf [rest of encrypted string omitted]",
"encrypted":true

 

Then just had to change my ‘router’ to decrypt the incoming data if encryption was detected.

else if(action == 'pizza')
{
	//pizza request contains encrypted info. Decrypt and send to function
	var pizzaRequestData = new Object();
	
	if(parsedContent.encrypted)
	{
		console.log('Encryped Payload Detected. Decrypting Containted Data');
		
		pizzaRequestData = JSON.parse(decrypt(parsedContent.data));
		
		console.log('Decryption complete');

		responseObject.message = 'Ordering Pizza!';
		
		console.log(responseObject.message);
		//because order pizza is async the result data comes in a callback
		orderPizza(pizzaRequestData,function(data){
			responseObject.pizzaRequest = data;
			console.log(data);
			sendResponse(response,responseObject);
			return;
		});
		}
	else
	{
		console.log('Un-encrypted pizza order detected. Skipping');
	}
}

After that I just had to use the encrypt function to generate an encrypted version of my pizza request data, update the IFTTT recipe with the new request and that’s it! Now my CC information is safely encrypted and I don’t really have to worry about it getting intercepted. Yay security.


Mimicking callback functions for Visualforce ActionFuncitons

Hey everyone. So I’ve got a nifty ‘approach’ for you this time around. So let me give you a quick run down on what I was doing, the problem I encountered and how I decided to solve it using what I believe to be a somewhat novel approach. The deal is that I have been working on a fairly complicated ‘one page’ app for mobile devices. What I decided to do was have one parent visualforce page, and a number of components that are hidden and shown depending on what ‘page’ the user is on. This allows for a global javascript scope to be shared between the components and also for them to have their own unique namespaces as well. I may cover the pros and cons of this architecture later.

The issue I started to have, is that I wanted some action functions on the main parent container page to be used by the components in the page. That’s fine, no problem there. The issue becomes the fact that since actionFunctions are asynchronous, and do not allow for dynamic callback functions anything that wants to invoke your actionFunction is stuck having the same oncomplete function as all the functions that may want to invoke it. So if component A and component B both want to invoke ActionFunctionZ they both are stuck with the same oncomplete function, and since it’s async there is no good way to know when it’s done. Or is there?

My solution to this problem doesn’t use any particularity amazing hidden features, just a bit of applied javascript knowledge. What we are going to do is create a javascript object in the global/top level scope. That object is going to have properties that match the names of action functions. The properties will contain the function to run once the action function is complete. Then that property will be deleted to clean up the scope for the next caller. That might sound a little whack. Here let’s check an example.

    <style>
        #contentLoading
        {
            height: 100%;
            width: 100%;
            left: 0;
            top: 0;
            overflow: hidden;
            position: fixed; 
            display: table;
            background-color: rgba(9, 9, 12, 0.5);  
              
        }
        #spinnerContainer
        {
            display: table-cell;
            vertical-align: middle;        
            width:200px;
            text-align:center;
            margin-left:auto;
            margin-right:auto;
        }

        div.spinner {
          position: relative;
          width: 54px;
          height: 54px;
          display: inline-block;
        }
        
        div.spinner div {
          width: 12%;
          height: 26%;
          background: #fff;
          position: absolute;
          left: 44.5%;
          top: 37%;
          opacity: 0;
          -webkit-animation: fade 1s linear infinite;
          -webkit-border-radius: 50px;
          -webkit-box-shadow: 0 0 3px rgba(0,0,0,0.2);
        }
        
        div.spinner div.bar1 {-webkit-transform:rotate(0deg) translate(0, -142%); -webkit-animation-delay: 0s;}    
        div.spinner div.bar2 {-webkit-transform:rotate(30deg) translate(0, -142%); -webkit-animation-delay: -0.9167s;}
        div.spinner div.bar3 {-webkit-transform:rotate(60deg) translate(0, -142%); -webkit-animation-delay: -0.833s;}
        div.spinner div.bar4 {-webkit-transform:rotate(90deg) translate(0, -142%); -webkit-animation-delay: -0.75s;}
        div.spinner div.bar5 {-webkit-transform:rotate(120deg) translate(0, -142%); -webkit-animation-delay: -0.667s;}
        div.spinner div.bar6 {-webkit-transform:rotate(150deg) translate(0, -142%); -webkit-animation-delay: -0.5833s;}
        div.spinner div.bar7 {-webkit-transform:rotate(180deg) translate(0, -142%); -webkit-animation-delay: -0.5s;}
        div.spinner div.bar8 {-webkit-transform:rotate(210deg) translate(0, -142%); -webkit-animation-delay: -0.41667s;}
        div.spinner div.bar9 {-webkit-transform:rotate(240deg) translate(0, -142%); -webkit-animation-delay: -0.333s;}
        div.spinner div.bar10 {-webkit-transform:rotate(270deg) translate(0, -142%); -webkit-animation-delay: -0.25s;}
        div.spinner div.bar11 {-webkit-transform:rotate(300deg) translate(0, -142%); -webkit-animation-delay: -0.1667s;}
        div.spinner div.bar12 {-webkit-transform:rotate(330deg) translate(0, -142%); -webkit-animation-delay: -0.0833s;}
    
         @-webkit-keyframes fade {
          from {opacity: 1;}
          to {opacity: 0.25;}
        }    	
	</style>
	
		var globalScope = new Object();
		
		function actionFunctionOnCompleteDispatcher(functionName)
		{
			console.log('Invoking callback handler for ' +functionName);
			console.log(globalScope.actionFunctionCallbacks);
			
			if(globalScope.actionFunctionCallbacks.hasOwnProperty(functionName))
			{
				console.log('Found registered function. Calling... ');
				console.log(globalScope.actionFunctionCallbacks.functionName);
				globalScope.actionFunctionCallbacks[functionName]();
				delete globalScope.actionFunctionCallbacks.functionName;
			}
			else
			{
				console.log('No callback handler found for ' + functionName);
			}    
		}         
		
		function registerActionFunctionCallback(functionName, callback)
		{
			console.log('Registering callback function for ' + functionName + ' as ' + callback);
			globalScope.actionFunctionCallbacks[functionName] = callback;
			
			console.log(globalScope.actionFunctionCallbacks);
		} 
		
		function linkActionOne(dataValue)
		{
			registerActionFunctionCallback('doThing', function(){
				console.log('Link Action Two was clicked. Then doThing action function was called. Once that was done this happened');
				alert('I was spawened from link action 1!');
			});		
			
			doThing(dataValue);
		}
		
		function linkActionTwo(dataValue)
		{
			registerActionFunctionCallback('doThing', function(){
				console.log('Link Action Two was clicked. Then doThing action function was called. Once that was done this happened');
				alert('I was spawened from link action 2!');
			});		

			doThing(dataValue);
		}

		function loading(isLoading) {
			if (isLoading) 
			{            
				$('#contentLoading').show();
			}
			else {
				$('#contentLoading').hide();
			}	
		}		
    
	
	<apex:form >
		<apex:actionFunction name="doThing" action="{!DoTheThing}" reRender="whatever" oncomplete="actionFunctionOnCompleteDispatcher('doThing');">
			<apex:param name="some_data"  value="" />
		</apex:actionFunction>
		
		<apex:actionStatus id="loading" onstart="loading(true)" onstop="loading(false)" />
	
		<a href="#" onclick="linkActionOne('Link1!')">Link One!</a>
		<a href="#" onclick="linkActionTwo('Link2!')">Link Two!</a>

		
id="contentLoading" style="display:none">
id="spinnerContainer">
class="spinner">
class="bar1">
class="bar2">
class="bar3">
class="bar4">
class="bar5">
class="bar6">
class="bar7">
class="bar8">
class="bar9">
class="bar10">
class="bar11">
class="bar12">
</div> </div> </div> </apex:form>

So what the hell is going on here? Long story short we have two links which both call the same actionFunction but have different ‘callbacks’ that happen when that actionFunction is complete. I was trying to come up with a more interesting example, but I figured I should keep it simple for sake of explanation.  You click link one, the doThing action is called. Then it calls the actionFunctionOnCompleteDispatcher function with it’s own name. That function looks to see if any callbacks have been registered for that function name. If so, it is called. If not, it just doesn’t do anything. Pretty slick eh? You may be wondering why I included all that code with the action status, the loading animation, the overlay and all that. Not really relevant to what we are doing right (though the animation is cool.)? The answer to that is (other than the fact you get a cool free loading mechanism), this approach as it stands will start to run into odd issues if you users clicks link2 before link1 has finished doing it’s work. The callback function registered by 2 would get called twice. Once the call to doThing from link1 one its going to call whatever function is registered, even if that means the click from link2 overwrote what link1 set.  I am thinking you could probably get around this by making the property of the global object an array instead of just a reference to a function. Each call would just push it’s requested callback into the array and then as they were called they would be removed from the array, but I haven’t played with this approach yet (‘I’m pretty sure it would work, I’m just too lazy and tired to write it up for this post. If there is interest I’ll do it later). In any case putting up a blocking loading screen while the action function does its work ensures that the user cannot cause any chaos by mashing links and overwriting callbacks.

The thing that is kind of cool about this which becomes clear pretty quick is that you can start ‘chaining’ callbacks. So you can have a whole series of action functions that all execute in sequence instead of just running async all over the place. So you can do things like this. Also make note of the commenting. The thing about callbacks is you can quickly end up ‘callback hell’ where it gets very difficult to track what is happening in what order. So I at least attempt to label them in an effort to stem the madness. This is just a quick copy paste from the thing I’m actually working on to give you a flavor of how the chaining can work.

//once a project has been created we need to clear out any existing temp record, then set the type of the new/current temp record as being tpe. Then finally
//we have to set the project Id on that temp record as the one we created. Then finally we can change page to the select accounts screen.
VSD_SelectProject.addNewTpeComplete = function()
{

	//order 2: happens after clearTempRecord is done 
	//once the existing temp record has been cleared and a new one is created, get a new temp record and set the type as tpe
	registerActionFunctionCallback('clearTempRecord', function(){
		setRequestType('tpe');
	});
				
	//order 3: happens after setRequestType is done 
	//once set request type is done (which means we should now have a temp record in the shared scope, then call set project id
	registerActionFunctionCallback('setRequestType', function(){
		setProjectId('{!lastCreatedProjectId}');
	});

	//order 4: happens after setProjectId is done 
	//once set project id is called and completed change the page to the new_pcr_start (poorly named, it should actually be called select_accounts
	registerActionFunctionCallback('setProjectId', function(){
		setTitle('New TPE Request');
		setSubHeader('Select Accounts');
					
		changePage('new_pcr_start');
	});
	 
	//order 1: happens first. Kicks off the callback chain defined above.                                                
	clearTempRecord();                
}

Anyway, I hope this might help some folks. I know it would be easy to get around this issue in many cases by just creating many of the ‘same’ actionFunction just with different names and callbacks but who want’s dirty repetitive code like that?

Tune in next time as I reveal the solution to an odd ‘bug’ that prevents apex:inputFields from binding to their controller values. Till next time!

-Kenji


Even more Automation with Tasker and Geofencing

So if you saw my last blog, you know I spent some time writing a small application that would log into my home security providers website, then lock and unlock my door on a timed schedule. If you haven’t read it, check it out at https://iwritecrappycode.wordpress.com/2015/10/21/automating-things-that-should-already-be-automated-with-selenium-and-node-js to get some context. While that was great and all, I figured I could do more with it. I mean timed scheduled are great, but I do occasionally leave the house so you know what would be really cool? If my door automatically locked when I left, and unlocked itself again when I returned home. I figured it should be a reasonably simple process to make my script respond to remote commands, and trigger my phone to send such commands when leaving and entering an area. Turns out I was mostly right. At it’s core its pretty easy, but there are a lot of moving parts and places to make mistakes, which I did make plenty of (it’s crazy how often as a programmer I type the wrong numbers in somewhere).

So obviously the first part of this challenge was going to be to setup my script to respond somehow to external requests. Being written in Node.js it made sense that it respond to web requests. I figured the requester should provide the username and password to login to the security website with, as well as a desired action of either lock or unlock. I decided to make it respond to POST requests just to reduce the possibility of some web spider hitting it and by some fluke making it do something (I also put the service on a different port, but we’ll cover that later). So the service listens for post requests with some JSON encoded data in the request body and then makes the request to alarm.com. The code for that method is as follows.

//create a server
server = http.createServer( function(request, response) {

    console.log('Request received from ' + request.connection.remoteAddress);

	var responseObject = new Object();
	responseObject.message = 'Waiting';
	responseObject.status = 'OK';

	//listen for POST requests
    if (request.method == 'POST') {

		console.log("POST Request Received");

		//when we get the data from the post reqest
        request.on('data', function (data)
		{
			try
			{
				console.log('Raw Data: ' + data);
				
				//parse content in the body of the request. It should be JSON
				var parsedContent = JSON.parse(data);
				
				//read some variables from the parsed json
				var action = parsedContent['action'];
				var password = parsedContent['password'];
				var username = parsedContent['username'];

				responseObject.action = action;


				if(action == 'lock')
				{
					responseObject.message = 'Sent Lock Request';
					console.log('Locking Door!');
					
					//because toggledoor is async the result data comes in a callback
					toggleDoor(username,password,true,function(data){
						responseObject.lockRequestData = data;
						sendResponse(response,responseObject);
						return;
					});

				}
				else if(action == 'unlock')
				{
					responseObject.message = 'Send Unlock Request';
					console.log('Unlocking Door!');
					//because toggledoor is async the result data comes in a callback
					toggleDoor(username,password,false,function(data){
						responseObject.lockRequestData = data;
						sendResponse(response,responseObject);
						return;
					});

				}
				else
				{
					console.log('Invalid Post Acton ' + action);
					responseObject.message = 'No method defined with name: ' + action;
					sendResponse(response,responseObject);
				}
			}
			catch(exception)
			{
				responseObject.message = 'Error: ' + exception.message;
				console.log(exception);
				sendResponse(response,responseObject);
			}
		});
    }
	else
	{
		console.log('Non post request. Ignoring.');
		responseObject.message = 'Please use post request';
		sendResponse(response,responseObject);
	}


}).listen(PORT, function(){
    //Callback triggered when server is successfully listening. Hurray!
    console.log("Server listening on: http://localhost:%s", PORT);
});

So now with a server that listens, I need to allow requests from the outside world to come in and actually get to my server. This is where my old days of networking came in handy. First I decided I’d like to have a domain name to reach my server instead of just an IP address (maybe I can setup dynamic DNS later so if my home IP address changes, my registar it automatically updated with the new one and will change my zone file records). I know this is more of a programming blog, but just a basic bit of networking is required. First I headed over to GoDaddy and grabbed myself a cheap domain name. Then changed the DNS settings to point the @ record at my home IP address.

Simple DNS Config

Now going to my domain would send traffic to my home, but of course it would be stopped at my firewall. So I had to setup port forwarding in my router. This is where I was able to change from using the web traffic port 80 to my custom port to help obscure the fact that I am running a web server (as well as having to change the listening port in the Node script). Of course I decided to turn off DHCP on the machine hosting this and give it a static IP address. I would have used a reservation but my crappy router doesn’t have that option. Lame.

port config

After some fiddling with settings for a bit my service was now responding to outside traffic. The only bummer part is that my domain name does not work within my network because again my router is crappy and doesn’t support NAT loopback, so it doesn’t know how to route the request to the right machine internally. I could modify my hosts file or setup my own DNS server but it’s not worth it seeing as this service is only useful when I’m outside my own network anyway.

Now the trickiest part, I needed to find a way to make my phone detect when I entered or left a given geographic region and when it detected that, send the specially crafted POST request to my server. I knew of two options off the top of my head that I had heard of but never played with. The first one I tried was a service called ‘If this then that’ or IFTTT for short. While the website and app are very sleek and they make configuration of these rules very easy, there was one problem. It just didn’t work. No matter what I tried, what recipes I configured nothing worked. Not even when triggered manually would it send the POST request. After playing with that for a bit, I decided to give up and give the other app I had heard of a shot. Tasker.

So if IFTTT is the mac of task automation (sleek, easy, unable to do the most basic things). Then Tasker is Linux. It’s extremely powerful, flexible, a bit difficult to understand and not much to look at. It does however have all the features I needed to finish my project. I ended up buying both Tasker (its like 3 bucks) and a plugin for it called AutoLocation. You see Tasker by itself is fairly powerful but it allowes for additional plugins to perform other actions and gather other kinds of data. AutoLocation allowed me to easy configure a ‘geofence’ basically a geographic barrier upon passing through which you can trigger actions. So I configured my geofence and then imported that config data into Tasker. Then it was simple a matter of created two profiles. One for entering the area and one for leaving it.

GeoFence Takser Profiles

I also added a simple vibrate rule so that my phone will buzz when either of the rules trigger so I know the command was sent. Later that night when I headed off to the store, that little buzz was the sweet vibration of success. I may or may not have yelled for joy in my car and frightened other motorists. I hope perhaps this post might inspire you to create own you crazy automation service. With the combination of selenium, node, tasker, and a bit of networking know how it’s possible to create all kinds of cool things.  If you’d like to download the source code for my auto lock program you can grab it below (I don’t really feel like making a git project for something so small. Also I realize its not the best quality code in the world, it was meant to be a simple script not a portfolio demo).

Download AutoLock Source

Till next time!
-Kenji


Automating Things That Should Already Be Automated With Selenium and Node.js

So I had invited my parents over a while back, and it came up that I don’t often lock my front door. Of course being good parents they chided me about it saying that I should really do so. I really have no excuse because I even have an app that allows me to to it remotely (yay home automation) but the be honest I’m just forgetful when it comes to things like that. However I decided to heed their warning and do something about it. I decided if I was going to get diligent about locking my door, I couldn’t be the one in charge of actually doing it, I’d have to make a computer do it. The problem is, that while my home security provider does offer a web app, and a phone app for locking the door and a very basic ‘rule’ system (arm the panel when the door is locked, vice versa) there are no time based controls, so I’d still have to actually do something. Totally unacceptable.

After some investigation I found as I had figured that my security provider does not offer any sort of API. Nor would it be easy to try and replicate the post request that is send from the app to trigger the lock door command due to numerous session variables and cookies and things include unique to each login session. Nope, if I was going to automate this it seemed like I’d actually have to interact with the browser as much as the thought displeased me (I’m all for dirty hacks, but c’mon). At first I looked at python for a solution, but as seems to often be the case with python every discussion of a solution was disjointed with no clear path and generally unsatisfactory (sorry python). Instead I turned to Node to see what potential solutions awaited me there. After a bit of looking around I found Selenium for Node. While it’s obvious and stated focus was on automated web app testing, not intentional browser automation scripts I could see no reason it wouldn’t work.

Quickly I spun up a new Node project and used NPM to grab the Selenium package (even after many uses NPM still feels like some kind of magic after manually handling javascript libraries for so long). Followed the guide to getting the Selenium web drivers to work, which at first seemed a bit odd having to install executable on my system for a javascript package but it makes perfect sense in retrospect. After finding a basic Selenium tutorial I was ready to attempt to get my script to login to alarm.com’s web page. First I had to get the names of the inputs I wanted Selenum to fill. Of course chrome makes this easy, just right click, inspect element and snag the names of the inputs.

Finding the required name property is easy.

Finding the required name property is easy.

 

Then simply tell Selenium to populate the boxes and click the login button.

 

var driver = new webdriver.Builder().
withCapabilities(webdriver.Capabilities.firefox()).
build();
   
driver.get('https://www.alarm.com/login?m=no_session&ReturnUrl=/web/Automation/Locks.aspx');
driver.findElement(webdriver.By.name('ctl00$ContentPlaceHolder1$loginform$txtUserName')).sendKeys(username);
driver.findElement(webdriver.By.name('txtPassword')).sendKeys(password);
driver.findElement(webdriver.By.name('ctl00$ContentPlaceHolder1$loginform$signInButton')).click();

Thankfully just through dumb luck when creating this my session had timed out and I found out the login page accepted a return url parameter that it would direct the browser to after successful login. So now, if the login goes smoothly, the browser should be at the screen where I can control the locks. A different button is used to lock or unlock the door and only one is visible at a time. So writing my function in such a way that it accepted boolean ‘lock’ (where false would be unlock) param and then failing if it’s not able to find the button is a safe way to ensure I don’t unlock the door when I mean to lock it and vice versa.

driver.wait(function() {
	return driver.getTitle().then(function(title) {
		console.log('Toggling Door Status');
		if(lock)
		{
			driver.findElement(webdriver.By.name('ctl00$phBody$summaryRepeater$ctl00$lockButton')).click();
			lockResult.message = 'Lock request sent!';
		}
		else
		{
			driver.findElement(webdriver.By.name('ctl00$phBody$summaryRepeater$ctl00$unlockButton')).click();	
			lockResult.message = 'UnLock request sent!';					
		}
		lockResult.success = true;
		

		driver.quit();

		return lockResult;
	});
}, 1000);

 

I don’t know Selenium super well yet, but it seems that after the login button is clicked the driver waits until it can retrieve the title of the page (which is a easy way to tell the page has at least somewhat loaded) and when it has then run the inner logic for clicking the proper button.  Honestly I’m not totally sure, but it works and that’s the important thing 😛

I was actually a bit shocked when my little function worked. Calling it with the proper username and password actually made the door lock a few moments later, much to my dogs surprise as he sat napping in the living room (the little motor on that lock is kind of loud). Now the next peice of this puzzle was to invoke that function on a timer system. Locking the door at say, 11:00pm and unlocking at 8:00am. This turned out to require only your regular every day javascript, nothing fancy.

var lockHour = 23;
var unlockHour = 8;
var beenLockedToday = false;
var beenUnlockedToday = false;

var date = new Date();
var current_hour = date.getHours(); 
console.log('Checking current hour for lock status checks. Current hour is ' + current_hour  + ' Will automatically lock at ' + lockHour + ' and unlock at ' + unlockHour + ' listening on port ' + port);
	
function monitorLoop() {
	
	
	date = new Date();
	current_hour = date.getHours();       
	
	console.log('Checking local hour. It is ' + current_hour);
	
	if(current_hour >= lockHour && !beenLockedToday) 
	{
		console.log('Lock hour hit or passed and door has not been locked. Locking!!');
		toggleDoor(alarm_username,alarm_password,true);		
		beenLockedToday = true;
	} 
	if(current_hour == unlockHour && !beenUnlockedToday) 
	{
		console.log('Un-Lock hour hit!');
		toggleDoor(alarm_username,alarm_password,false);		
		beenUnlockedToday = true;
	}
	if(current_hour == 0)
	{
		console.log('Resetting lock status variables');
		beenLockedToday = false;
		beenUnlockedToday = false;		
	}

	setTimeout(monitorLoop,600000);

}

monitorLoop();

It’s just a function that is called via setInterval every hour. It checks the current hour against my two predefined lock and unlock times. I used a couple variables to track if the door has been locked or unlocked so if I reduce the time on the event loop it’s not attempting to lock/unlock the door every few minutes and wearing out the batteries on the motor. Obviously omitted from this is my alarm_username and alarm_password variables stored higher up in the script. With this event loop and the Selenium automation function I now have one less thing to worry about. Now if I could just find a Node.Js host that supported Selenium (damn you Heroku). So if anything, I’d say this is the take away: Don’t do manually what you can automate, and browser automation with Selenium is crazy easy. So easy that when it all worked I was almost disappointed that it seemed like I didn’t do anything.

Glorious event loop in action

Glorious event loop in action

Till next time!
-Kenji

Also, be sure to check out the addendum to this project in my next blog post where I added automatic operations with a geofence. https://iwritecrappycode.wordpress.com/2015/10/21/automating-things-that-should-already-be-automated-with-selenium-and-node-js/


Communication Between Visualforce Pages and Standard Pages Via Iframes

So I had what I thought was going to be a quick easy project recently, an idea spawned from some idea spit-balling at the office. Basically we wanted to create a page where you could queue up records, and it would show you the edit screen for them one at a time. Hit save, next record comes up in edit mode, save it, next record comes up in edit mode. Sounds a lot like the inline list view editor I know, but this would give you the full edit screen, ability to modify rich text areas, etc. I dunno, some other people wanted it. I decided it sounded fun and easy enough to build so I decided to take a crack at it.

I had a few internal conditions I wanted to satisfy as well. I wanted it to be light weight, fast, and not require much if any configuration to work for any object in Salesforce. It should be flexible, easy and straight forward. I didn’t want some huge apex controller backing it, required custom buttons or anything weird like that. This was a simple project and it should have a simple solution. The approach I decided to go with was to create a visualforce page that received a list of Ids to edit, it would then request the record for each of those ideas via an iframe. The iframe avoids having to run any messy queries, page layouts are respected, all that good stuff. Using some URL hacks I could tell it to load the record straight in edit mode (if you append a /e on the end of a record url it loads straight into edit mode), and even set the return URL (where the browser is directed after the user saves/cancels), seemed easy enough. Have the frame load the first record, set the return url to being the edit mode of the next record, and boom just follow that process until you exhaust the list. I forgot one crucial thing, well not so much forgot as ignored and hoped I could figure a way around it. Cross domain iframe security.

Let’s back up one second. If you are not aware, when using iframes if the parent frame and the child frame are both in the same domain you can use javacript to transverse them, inspect the child, share variables in the global JS scope, all that good stuff. If they are not on the same domain about the only thing you can do is load the frame, and tell when it’s source changes/reloads. That’s it. You can’t insepct it, you can’t modify it, you have extremely minimal control. Salesforce hosts visualforce pages and standard object pages on different domains. Bummer. This means I could not modify the behavior of the save button, and I couldn’t know if the user saved, cancelled, or anything about what they were doing in the frame. This means that after the got passed the second record (the first I could load into the frame directly, and then by setting the returnURL I could get them to go to the second one) I could no longer control where the user was going to easily. Sure I could change the src on the iframe itself but the only time I could do that was during a load event, which quickly gets you stuck in a loop. I know it sounds like it should be easy, but take a whack at it, I was unable to figure out an elegant solution after a couple hours. Bottom line, this wasn’t going to be as easy as I thought.

So here I was kind of stuck, I could load the first record and even set it to saving and returning to the second record in edit mode, but that was about it. The iframe source doesn’t change on record saves, and there isn’t any way to modify the behavior of the save button to return to the next record URL. I did come up with a hacky version that by detecting every other page load I could change the source to the assumed next record. EX the first record loads and the user saves. Code detects 2nd load event (which is now the detail view of the first record after the save) and changes the source to the next record. Waits for another reload and forwards them again. It worked, but it was hacky, unsatisfying and required waiting for the detail version of the page to load before forwarding the user onto the next edit mode. There had to be a better way, but I’d need to somehow modify the contents of the iframe.

I can’t say exactly how I remembered, but suddenly a few pieces of information kind of hit me all at once.
1) Sidebar components can run custom javascript and run off the core salesforce domain, not the visualforce domain.
2) HTML5 spec gives us a postMessage API to frames from different domains to pass information to each other.
3) Javascript can run arbitrary code using eval.

Could I stitch all this information together to build some kind of javascript proxy to run code on the core domain on behalf of my visualforce page? Indeed I could, and now I’ll show you how. Gather round; Kenji’s got a cool new trick to show you.

First you are going to need to enable custom sidebar components on all pages. Head to:

setup-> user interface -> check: Show Custom Sidebar Components on All Pages

Now create a narrow HTML homepage component. Slap this code in there. Yeah I know eval is bad, you can of course replace this with specific function calls if you want but I wanted to play around. Besides SF is a mostly secure environment.

<div id="jsProxy" style="width:100%;height:100px"></div>
<script>
    function receiveMessage(event) {
		var result = new Object();
		result.success = true;
		result.callback = event.data.callback;
		result.command = event.data.command;
		
        try {
            document.getElementById("jsProxy").innerHTML = 'JS proxy received command: ' + event.data.command;
            result.data = eval(event.data.command);
         
        } catch (ex) {
			result.data = ex.message;
			document.getElementById("jsProxy").innerHTML = 'Error running command: ' + ex.message;
        }
		event.source.postMessage(result,event.origin);
    }
    window.addEventListener("message", receiveMessage, false);
</script>

What we just did is created an event listener that listens for any postmessages coming in. It hands them off to our receiveMessage function. That will set the content of our div so we can see what got passed in (mostly for debugging) and then attempts to run the received message data as code. This received code is now running in the context of the core salesforce page, not within visualforce. With this we can modify elements on the standard object interface.

So now you might be saying, okay great, you can pass code to the page to run, but still, how are you going to change the returnURL? It’s too late to try and pass that into the URL by the time the page loads right? You are right, but thankfully Salesforce usually just takes URL parameters and jams them into hidden fields and uses them for whatever from there. So it’s just a matter of changing the value of the hidden ‘retURL’ field. So I pass to my function a bit of code that locates that field and modifies it to be the url of the next record in edit mode. Something like this.

    //where to go next
    var nextRecord = multiEdit.objectIds[parseInt(indexPosition+1,10)];
    
    //create command to pass to listener to run
    message = 'document.getElementById("retURL").value = "/'+nextRecord+'/e"';
    
    //create container variable so we could specify a callback if we wanted to
    var setReturnUrl = new multiEdit.postMessageData(message,null);
    
    //send the command
    iframeMessageWindow.postMessage(setReturnUrl,'*'); 
                       
    multiEdit.receiveMessage = function(event)
    {
        if(event.data.callback != null)
        {
            window.multiEdit[event.data.callback](event);
        }
        
    }
    
    multiEdit.postMessageData = function(command, callback)
    {
         this.command = command;
         this.callback = callback;   
    }
   

    window.addEventListener("message",multiEdit.receiveMessage, false);

Boom, now the code for changing the field is passed via the postMessage API to the homepage component that has the ability to run said code and operate on the DOM within the iframe. Pretty slick eh? With this technique the border between visualforce pages and standard pages has pretty much been smashed. Sure it needs a bit of refining, but overall I think this could allow for some pretty cool stuff.

BTW I’ll probably be releasing the utility that I wrote using this approach fairly soon. Till next time!


Salesforce Orchestra CMS Controller Extensions

So I’ve been working with Orchestra CMS for Salesforce recently, and for those who end up having to use it, I have a few tips.

1) If you intend on using jQuery (a newer version than the one they include) include it, and put it in no conflict mode. Newer versions of jQuery will break the admin interface (mostly around trying to publish content) so you absolutely must put it in no conflict mode. This one took me a while to debug.

2) While not official supported, you can use controller extensions in your templates. However the class, and all contained methods MUST be global. If they are not, again you will break the admin interface. This was kind of obvious after the fact, but took me well over a week to stumble across how to fix it. The constructor for the extension takes a cms.CoreController object. As an alternative if you don’t want to mess with extensions what you can do is use the apex:include to include another page that has the controller set to whatever you want. the included page does not need to have the CMS controller as the primary controller, so you can do whatever you want there. I might actually recommend that approach as Orchestra’s official stance is that they do not support extensions, and even though I HAD it working, today I am noticing it act a little buggy (not able to add or save new content to a page).

3) Don’t be araid to use HTML component types in your pages (individual items derived from your page template) to call javascript functions stored in your template. In fact I found that you cannot call remoting functions from within an HTML component directly, but you can call a function which invokes a remoting function.

So if we combine the above techniques we’d have a controller that looks like this

global class DetailTemplateController
{
    global DetailTemplateController(cms.CoreController stdController) {

    }

    @remoteAction
    global static list<user> getUsers()
    {
        return [select id, name, title, FullPhotoUrl from user ];
    }
}

And your  template might then look something like this

<apex:page id="DetailOne" controller="cms.CoreController" standardStylesheets="false" showHeader="false" sidebar="false" extensions="DetailTemplateController" >
	<apex:composition template="{!page_template_reference}">
		<apex:define name="header"> 
			<link href="//ajax.aspnetcdn.com/ajax/jquery.ui/1.10.3/themes/smoothness/jquery-ui.min.css" rel='stylesheet' />

			<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
			<script> var jqNew = jQuery.noConflict();</script> 
			<script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js"></script> 

			<script>
        	        var website = new Object();
			jqNew( document ).ready(function() {
				console.log('jQuery loaded');
			});

			website.buildUserTable = function()
			{
				//remoting request
				Visualforce.remoting.Manager.invokeAction(
					'{!$RemoteAction.DetailTemplateController.getUsers}', 
					function(result, event){
						if (event.type === 'exception') 
						{
							console.log(event.message);
						} 
						else 
						{
							var cols = 0;

							var tbl = jqNew('#bioTable > tbody');
							var tr;
							for(var i = 0; i < result.length; i++)
							{
								if(cols == 0){tr = jqNew('<tr></tr>');}                              

								var td = jqNew('<td></td>');

								var img = jqNew('<img class="profilePhoto">');
								img.attr('src',result[i].FullPhotoUrl);
								img.attr('title',result[i].Title);
								img.attr('alt',result[i].Name);
								img.data("record", result[i]);
								img.attr('id',result[i].Id);

								td.append(img);

								tr.append(td);

								if(cols == 2 || i == result.length-1){
									tbl.append(tr);
									cols = -1;
								}
								cols++;

							}

						}
					})			
			}
			</script>
		</apex:define>
		<apex:define name="body">
			<div class="container" id="mainContainer">
				<div class="pageContent">
					<div id="header">
						<apex:include pageName="Header"/>
						<div id="pageTitle">
							<cms:Panel panelName="PageTitle" panelController="{!controller}" panelheight="50px" panelwidth="200px"/>
						</div>
					</div>
					<div id="pageBody">
						<p>
							<cms:Panel panelName="PageContentArea" panelController="{!controller}"  panelheight="200px" panelwidth="400px" />
						</p>
						<div class="clearfloat"></div>
					</div>

					<!-- end .content --> 
				</div>
			</div>
			<div id="footer_push"></div>
			<div id="footer">
				<apex:include pageName="Footer"/>
			</div>
		</apex:define>
	</apex:composition>
</apex:page>

Then in our page we can add an HTML content area and include

<table id="bioTable">
	<tbody></tbody>
</table>
<script>website.buildUserTable();</script>

So when that page loads it will draw that table and invoke the website.buildUserTable function. That function in turns calls the remoting method in our detailTemplateController extension that we created. The query runs, returns the user data, which is then used to create the rows of the table that are then appended to the #bioTable’s body. It’s a pretty slick approach that seems to work well for me. Your mileage may vary, but at least rest assured you can use your own version of javascript, and you can use controller extensions, which I wasn’t sure about when I started working it. Till next time.


Salesforce Live Agent Review & Customization

So you are building a new website hosted on force.com and your boss says

‘Oh and we gotta have chat. Everyone has chat these days, we need it.’

Agree, disagree, doesn’t matter. You are doing it because rent is coming due and you can’t tell him that that idea is as bad as his comb-over (all purely hypothetical of course). So you start thinking about writing your own chat app because it sounds like fun (some UI options, push notifications, some cool chances to use javascript remoting maybe?), then realize you don’t have time for fun because this thing is due in like a week. So you frantically google around a bit and realize,

‘Wait a minute, Salesforce has it’s own native chat app “live agent”. That could probably do most of my work for me!’

Only question is can you hack at it enough to make it do what you need? Does it have a pre-chat form? Does it have a post chat survey? Does it save logs? How about a queue system? The short answer is yes. I was actually blown away at how much time and energy they put into the live agent chat. It can do pretty much everything any reasonable person would ask of it, and a few unreasonable things as well.  As a developer though you just want to get it up and running as fast as possible so you can play with all the bells and whistles right? Manuals are readme’s are for suckers, let’s just throw it on a page somewhere as a POC to make the boss man happy. So how do you go about doing that, what does the process look like? In a nutshell, it’s going to go like this.

1) Get the Salesforce liveagent trial for your org.
2) Log into Salesforce and go to customize->live agent.
3) Create a deployment, and a button
4) Paste the scripts they give you onto your webpage
5) Feel slightly disappointed that it was too easy and you didn’t get to derive any satisfaction from solving a problem yourself.

OH! Before I forget, I felt really dumb when I first set this up because I put it all in place and couldn’t figure out how to actually let an agent login to the chat. I kept seeing something about a console view, but the console didn’t say anything about chat. It turns out you just have to create a console view (you know like you do for cases or whatever) and in the lower right corner there is a chat login in. It’s all within Salesforce, there is no other service to authenticate to or anything, which is pretty sweet.

So now you show your boss, and he’s like ‘Yeah that’s cool, but we’d like to know who we are talking to via a pre-chat form, and if they don’t exist in our system it would be cool to create a lead for them. Also if nobody is online we should just redirect them to a contact us form or something’. Ah-hah finally something fun! So you probably saw while creating your chat button there was a lookup field to a pre-chat form, but didn’t really have anything to populate that with. Nor do you have any idea how to build one. Well it turns out Salesforce actually has a pretty robust API centered around their chat, and a couple ways to pass information to the agent responding to the chat. I’m going to focus on the ‘simple’ form based approach since I haven’t used their javascript API they offer. So this pre-chat form can actually perform lookups, pass data, save information into the chat record itself, it’s pretty wild, but a little confusing. So first check out the pre-chat form sample Salesforce provides, it gives a good basic understanding of how it works.

Salesforce Pre-Chat Form Sample

You can see that you create fields, then some hidden fields take the values of the user entered fields and run queries or pass that info along to the console. You can populate fields on the chat record by creating ‘fields’ that look like this

<input type=”hidden” name=”liveagent.prechat.save:Email”  value=”Provided_Email__c” />

That says, take the value of the field called liveagent.prechat.Email from this form and save it into the Provided_Email__c field on the chat history object. Of course you could reference another hidden field in that name attribute and use code to set the value of that other hidden field allowing you to pass in basically whatever you like to the chat history. You can create custom fields and pass values to them, as well as passing values to the standard fields too.

But now we need to solve our issue of lookuping up the contact or lead based on email, and if one doesn’t exist creating one on the fly and returning that.  There are a few ways you could take this, but since I love javascript I decided to go with the javascript remoting approach. The user is going to enter their name and email, when they click submit instead of actually submitting the form a remoting function is going to run that runs our query and returns the result, or creates one. With that data we will populate the contactid or leadid field (depending what kind of record it is) and pass that along to the console by using javascript to submit the form once the function has run. Additionally using the ability to detect if there are agents online or not, we can change the behavior of the buttons (well actually we just show and hide different buttons) to send the user elsewhere if there is nobody online. It looks something like this.

<apex:page showHeader="false" controller="PreChatController">

<apex:variable var="deploymentId" value="572c0000000CaRW" />
<apex:variable var="orgId" value="00Dc0000001M6Ix" />
<apex:variable var="buttonId" value="573c0000000CaSe" />

<!-- This script takes the endpoint URL parameter passed from the deployment page and makes
it the action for the form -->
<script type='text/javascript' src='https://c.la7cs.salesforceliveagent.com/content/g/js/29.0/deployment.js'></script>

<script type='text/javascript'>
    liveagent.init('https://d.la7cs.salesforceliveagent.com/chat', '{!deploymentId}', '{!orgId}');
</script>

<script type="text/javascript">
    (function() 
    {
        function handlePageLoad()
        {
            var endpointMatcher = new RegExp("[\\?\\&]endpoint=([^&#]*)");
            document.getElementById('prechatForm').setAttribute('action',
            decodeURIComponent(endpointMatcher.exec(document.location.search)[1]));
        } 
        if (window.addEventListener) 
        {
            window.addEventListener('load', handlePageLoad, false);
        } 
        else 
        { 
            window.attachEvent('onload', handlePageLoad, false);
        }
    })();

    if (!window._laq) { window._laq = []; }

    window._laq.push(function()
    {
        liveagent.showWhenOnline('{!buttonId}', document.getElementById('prechat_submit'));
        liveagent.showWhenOffline('{!buttonId}', document.getElementById('liveagent_button_offline_{!buttonId}'));
    });

    function getLeadOrContact()
    {
        console.log('Getting lead or contact');
        var emailAddr = document.getElementById('email').value.trim();
        var fname = document.getElementById('name').value.trim();
        var phone = document.getElementById('phone').value.trim();

        try
        {
            Visualforce.remoting.Manager.invokeAction(
                '{!$RemoteAction.PreChatController.findLeadOrContactByEmail}', 
                fname,
                emailAddr,
                phone, 
                function(result, event)
                {
                    if (event.status) 
                    {
                        console.log(result);
                        if(result.Id.substring(0,3) === '003')
                        {
                            document.getElementById('contactid').value = result.Id;
                        }
                        else if(result.Id.substring(0,3) === '00Q')
                        {
                            document.getElementById('leadid').value = result.Id;
                        }
                        document.forms["prechatForm"].submit();

                        return true;
                    } 
                }, 
                {escape: false}
            );
        }
        catch(ex)
        {
            alert(ex.message);
            console.log(ex);
            return false;
        }
        return false;
    }   

</script>
<style>
body
{
    background-color:#f4f4f4;
}
#chatFormDiv
{
    width:200px;
    text-align:center;
    padding:5px;
}
#chatHeader
{
    color:#6d6d6d;
    font-size:18px;
    font-weight:bold;
}
label
{
    width:150px;
    font-weight:bold;
}
input[type=text], textarea
{
    width:200px;
    background: #f3f3f3; /* Old browsers */
    background: -moz-linear-gradient(top, #f3f3f3 0%, #ffffff 100%); /* FF3.6+ */
    background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#f3f3f3), color-stop(100%,#ffffff)); /* Chrome,Safari4+ */
    background: -webkit-linear-gradient(top, #f3f3f3 0%,#ffffff 100%); /* Chrome10+,Safari5.1+ */
    background: -o-linear-gradient(top, #f3f3f3 0%,#ffffff 100%); /* Opera 11.10+ */
    background: -ms-linear-gradient(top, #f3f3f3 0%,#ffffff 100%); /* IE10+ */
    background: linear-gradient(to bottom, #f3f3f3 0%,#ffffff 100%); /* W3C */
    filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#f3f3f3', endColorstr='#ffffff',GradientType=0 ); /* IE6-9 */    
    border-color: #dedede;
    border-top-color: #d3d3d3;
}
textarea
{
    height:140px;
}
.chatStatusDiv
{
    display:none;
}
</style>

<div id="chatFormDiv">
    <img src="{!URLFOR($Resource.BeaconWebsite,'img/chatIconSmallGrey.png')}" /> <span id="chatHeader">Chat</span><br/>
    <hr />

    <form method='post' id='prechatForm' onsubmit="return false;" action="https://15af.la7cs.salesforceliveagent.com/content/s/chat?language=en_US#deployment_id={!deploymentId}&org_id={!orgId}&button_id={!buttonId}">

    <input type='text' name='liveagent.prechat.name' id='name' placeholder="Your Name" required="required"/><br />

    <input type='text' name='liveagent.prechat:Email' id='email' placeholder="Email Address" required="required" /><br />

    <input type='text' name='liveagent.prechat:Phone' id='phone' placeholder="Phone" required="required" /><br />

    <textarea name='liveagent.prechat:Body' id='body' placeholder="Message" required="required" ></textarea><br />

    <input name="liveagent.prechat.buttons" value="{!buttonId}" type="hidden" /><br />

    <!-- Creates an auto-query for a matching Contact record’s Email field based on the
    value of the liveagent.prechat:Email field -->
    <input type="hidden" name="liveagent.prechat.query:Email" value="Contact,Contact.Email" />

    <!--- populate fields ---->
    <input type="hidden" name="liveagent.prechat.query:Email" value="Lead,Lead.Email" />
    <input type="hidden" name="liveagent.prechat.save:Email"  value="Provided_Email__c" />
    <input type="hidden" name="liveagent.prechat.save:name"   value="Provided_Name__c" />
    <input type='hidden' name='liveagent.prechat:ContactId'   value='' id='contactid'/>  
    <input type="hidden" name="liveagent.prechat.save:ContactId" value="Contact" />

    <input type='hidden' name='liveagent.prechat:LeadId' id='leadid' />  
    <input type="hidden" name="liveagent.prechat.save:LeadId" value="Lead" />

    <!--- Button that shows up if someone is online --->    
    <img   src="{!URLFOR($Resource.BeaconWebsite,'img/chatButton.png')}" 
           alt="Submit Form" 
           onclick="getLeadOrContact()"
           id='prechat_submit'
           style="display: none; border: 0px none; cursor: pointer; float: left;"/>

    <!--- Button that shows up if nobody is online --->
    <img id="liveagent_button_offline_{!buttonId}" 
         style="display: none; border: 0px none; cursor: pointer; float: left;" 
         src="{!URLFOR($Resource.BeaconWebsite,'img/chatButton.png')}" 
         onclick="alert('nobody online, put your redirect here');"/>

    </form>
</div>
</apex:page>

And the Apex Controller looks like this

global class PreChatController
{

    @remoteAction
    global static sobject findLeadOrContactByEmail(string name, string email, string phone)
    {
        sObject returnObject; //the id that will store the contact or lead this chat is related to
        //first we should see if there is a contact with this email
        list<contact> contacts = [select accountid, name, id, email from contact where email = :email limit 1];
        if(!contacts.isEmpty())
        {
            return contacts[0];
        }

        //if there is no contact, then lets look for a lead instead. Yeah we could combind both queries into a single SOSL search, but the code for that doesn't
        //end up being much cleaner when you account for having to figure out what record to return when you have a list of lists.
        else
        {
            list<lead> leads = [select name, id, email from lead where email = :email limit 1];
            if(!leads.isEmpty())
            {
                return leads[0];
            }
            else
            {            
                lead thisLead = new lead();
                string[] nameParts = name.split(' ');

                thisLead.firstname = nameParts.size() > 1 ? nameParts[0] : ''; //if name parts contains more than one element that means we likely got a full name and the first part is the firstname. Otherwise nothing
                thisLead.lastname =  nameParts.size() > 1 ? nameParts[1] : nameParts[0]; //if name parts is greater than 1 then use the 2nd element as the lastname. Otherwise use the first element
                thisLead.phone = phone;
                thisLead.email = email;
                thisLead.company = name;
                thisLead.leadSource = 'Web Site';

                insert thisLead;
                return thisLead;
            }
        }
    }
}
Chatone

The user is asked to enter their information to connect with an agent.

chatTwo

When the agent is alerted that there is a person waiting to chat they automatically get a ‘screen pop’ informing them of the contact details

chatThree

The full contact or lead record is available to the agent making all their persons information instantly available. Much nicer than having to ask them a million questions.

Now when a user uses your chat form they will have to fill in their email, name and phone. Using that we can either locate them if they already exist, or create them on the fly and pass that information along to the agent. Pretty slick eh? One small bug that I dislike and currently do not know how to fix is that even though the contactId/leadId is passed to the console those fields are not populated visually. So your agent doesn’t see the lookups populated, even though they are. I think for now it’s just a training point to tell them. When the chat is saved and closed the transcript will be related to the contact/lead you just can’t see it on this screen. Weird I know.

Anyway I hope this helps getting you started and maybe even finished implementing Live Agent. It’s pretty cool and has a lot of features/power…. way more than I would have guessed. Best of all it makes you look like some kind of development wizard when you get go from request to implementation in like a few hours, so now you can get back to reddit… until the next request comes.


Saltybot – A descent into salty, salty madness

*Article below was written pertaining to Saltybat as it was in September 2013, I have no idea the current standing of the website or if the techniques below are still applicable/required (looking now it seems that the fighter names are available right next to the bet buttons, so the whole needing an OCR scanner to extract the names is no longer required. Bummer that made me feel so smart too)

I. CAN’T. STOP. SALTYBETTING.

I don’t even care about the bucks, it’s about solving the problem. I know that somewhere in the chaos there is data that will allow me to get every bet correct and ‘solve’ the problem. But wait, let me back up a step. First, what the hell is Saltybet? It’s an online video feed of your favorite characters new and old fighting it out controlled by computer AI. You bet on the matches and get ‘saltybucks’ if you win based on a weighted odds system. Before they duke it out both characters stand on screen for about 45 seconds allowing you to consider whom to bet on. After that window betting closes the screen blanks out for a moment, then the fight begins. At that point all you can do is sit and watch (and bitch in the chat window). No you can’t do anything with the money and there is very little rhythm or reason to what makes a good character (one of the best, if not the best currently is an interpretation of Ronald McDonald). It’s pretty addictive to watch and I’ve been having a good time just keeping it on my spare monitor during my work day.

THEN

IT

HIT

ME

I could write a small program, just a little javascript bookmarklet that could take the names of the fighters and using past data tell me who is most likely to win. Hell maybe I could even automate some of it. Overcome with excitement I pushed the major technical challenges out of my mind so not to kill my motivation. I just wanted to try to see what I could do. So my last weeks obsession began.

The Bot

First off, I just got this working no more than like 45 minutes ago so I’m pretty excited about it. That does mean that though some of the code I show you and approaches I use may not be the best, as they are mostly just POC to see if my architecture works. I don’t intent to release this final product so it may never get totally cleaned up. That aside, please find below my decent into madness as I write one of the most complicated cobbled` together things I’ve ever even heard of.

I needed data. A lot of it. For an analytic engine you need data points. Thankfully if you are willing to pitch in a few bucks for server costs you get access to all the characters win loss record and access to the results of every match you have bet on. So not only could you calculate win percentage if you know who various characters won and lost against you could implement a rating system like they use in chess (ELO) or maybe even Glicko. The issue here though was that of course that data was not presented in any kind of API, it was just an HTML table. So the first challenge was trying to get that HTML data into some kind of actual data structure. After a little bit of ajax and jQuery magic I was able to convert the table into a javascript object keyed by the fighters name. I ran their name though a simple regex to remove spaces, make it lower case to make matching easier. I was making progress, now typing in the two fighters name would net me their win loss percents, which are powerful numbers but of course don’t tell the whole story (a person with a 1-0 record might have a 100% win loss, but that isn’t nearly as good someone with a 30-5 record even though they’d have a lower win percent).

Next up was trying to implement a real rating system. I know there are systems out there that could tell you the true strength of a player based on their previous matches and who they won and lost against. Chess ratings implement this kind of system as well as various competitive online games. I knew that something like that could go a long way toward getting me to real odds. The problem is that you can only see the record of who a fighter won and lost against if you bet on that match. There was no way I’d have time to sit around and bet on every match, nor did I want to. I knew I’d have to setup some kind of automated betting to just put in small bets for me to gather data while I was doing other things. First though, I wanted to get the rating system in place so as data came in I could see it affect the results of the equation. After some research and consideration I decided on the ELO system (mostly because I am a bit familiar with it, and its fairly easy to implement and I found some sample code so I didn’t have to write it myself XD ). The basic idea is that every character starts at a rating of 1200, then based on who they win or lose against their rating changes. The more matches they go through the higher the confidence of the result. The simple function for calculating the change in score looks like this.

function CalculateEloRatingChange(player1Score,player2Score,result,gamesPlayed) 
{ 

	var Elo1 = player1Score; 
	var Elo2 = player2Score; 
	var K = Math.round(800/gamesPlayed); 
	var EloDifference = Elo2 - Elo1; 
	var percentage = 1 / ( 1 + Math.pow( 10, EloDifference / 400 ) ); 
	var win = Math.round( K * ( 1 - percentage ) ); 
	var draw = Math.round( K * ( .5 - percentage ) ); 
	if (win > 0 ) win = "+" + win; 
	if (draw > 0 ) draw = "+" + draw; 

	if(result == 'win')
	{
		return parseInt(win,10);
	}
	else
	{
		return parseInt(Math.round( K * ( 0 - percentage ) ),10); 
	}
}

Like I said this was pulled from another source, so I’m not 100% certain on the logic used, but it seems solid and it’s been returning what seem to be reasonable numbers for me. So now all I had to do is pull the betting history table and iterate over it back start to finish, each time feeding the result of the match into this function and recording the new ELO score for that character. Of course since I only have access to the data for matches I have bet on, this is not perfect or absolute, but it is better than nothing and as I keep gathering more data it will just get more and more accurate (addendum: later on I started farming results from other players. Offering them access to the tool in exchange for their betting history which I could feed into my engine).

Now I was able to get a win percent and an ELO score. I was well on the way to having some meaningful data that could point me in the right direction. Both these facts left out something that I thought was pretty crucial though. If this EXACT matchup has happened before the results of that are likely to be repeated and should definitely be taken into consideration. So In the betting history I also decided to look to see if this same match had happened before. If so I initially just printed out a warning to my utility to let me know. I knew that that should have it’s own numerical meaning as well but I couldn’t find any formula like that online so i decided to brew my own. I really don’t have much a background in probability and stats or anything like that so I am really not sure about the weights that I assigned the various outcomes. Maybe someone with those skills could help me tweak this. Overall my scoring formula looks like this

function calculateProjectedWinner(player1Name,player2Name)
{
	//find players rating difference and record
	fighterCareers[player1Name].ratingDiff = fighterCareers[player1Name].eloScore - fighterCareers[player2Name].eloScore;
	fighterCareers[player2Name].ratingDiff = fighterCareers[player2Name].eloScore - fighterCareers[player1Name].eloScore;

	//calculate their win probabilities. The Elo system has it's own function for calculating win probability
	//based on scores, so I just use that as my 'baseline' probabilities. Then I modify it using my other data later on.
	fighterCareers[player1Name].eloWinProbability = parseInt(calculateEloWinOddsPercent(fighterCareers[player1Name].ratingDiff) * 100,10);
	fighterCareers[player2Name].eloWinProbability = parseInt(calculateEloWinOddsPercent(fighterCareers[player2Name].ratingDiff) * 100,10);

	//calculate custom their win probabilities starting at ELO
	fighterCareers[player1Name].computedWinProbability = fighterCareers[player1Name].eloWinProbability;
	fighterCareers[player2Name].computedWinProbability = fighterCareers[player2Name].eloWinProbability;

	//now we need to see if these two players have had any previous matches together. If so we iterate over them
	//and modify their win probabilities accordingly.
	var prevMatches = findPreviousMatch(player1Name,player2Name)		

	for(var i = 0; i < prevMatches.length; i++)
	{
		var winner = prevMatches[i].winner;
		var loser = prevMatches[i].loser;

		//we don't want to make their probability much higher than 95 because we can never be that sure and also
		//anything over 100 is totally meaningless. I decided a factor of 8 percent per win seems about decent. Maybe
		//it should be a little more? I don't know it's still something I'm kind of playing with. 
		if(fighterCareers[winner].computedWinProbability < .92)
		{
			fighterCareers[winner].computedWinProbability = fighterCareers[winner].computedWinProbability + 0.08;
		}
		if(fighterCareers[loser].computedWinProbability > .08)
		{
			fighterCareers[loser].computedWinProbability = fighterCareers[loser].computedWinProbability - 0.08;
		}
	}

	//their win loss percent can be a good statistic if it is composed of enough data points to be meaningful.
	//here is where I wish I had more prob and stats background because I really don't know how many matches it would
	//take for this percent to be actually significant. I'm guessing at 10, so I decided to go with that. If both chars
	//have more than 10 matches under their belt, then lets include their win loss percents in our calculation.
	if(fighterCareers[player1Name].total >= 10 && fighterCareers[player2Name].total >= 10)
	{
		//get the difference between the two win percents. So if we had p1 with 50 and p2 with 75 the difference is 25
		//yes I know ternaries are hard to read, but its cleaner than a stupid one line if statment. Just know that this
		//will return a positive amount that is the difference in win percent between the two.
		var winPercentDifference = fighterCareers[player1Name].winPercent > fighterCareers[player2Name].winPercent ? fighterCareers[player1Name].winPercent - fighterCareers[player2Name].winPercent : fighterCareers[player2Name].winPercent - fighterCareers[player1Name].winPercent;

		//multiple that difference by how confident we are (total number of matches) topping out at. So a number from 20 to 100
		var confidenceScore = fighterCareers[player1Name].total + fighterCareers[player2Name].total > 100 ? 100 : fighterCareers[player1Name].total + fighterCareers[player2Name].total;

		var adjustment = Math.round((winPercentDifference) * (confidenceScore/100)/2);

		//make the actual adjustments to the players probabilities
		console.log('Proposed modifying win perceny by +/- '+ adjustment);
		if(fighterCareers[player1Name].winPercent > fighterCareers[player2Name].winPercent)
		{
			fighterCareers[player1Name].computedWinProbability += adjustment;
			fighterCareers[player2Name].computedWinProbability += adjustment*-1;	
		}
		else
		{
			fighterCareers[player1Name].computedWinProbability += adjustment*-1;
			fighterCareers[player2Name].computedWinProbability += adjustment;			
		}
	}

	//find the winner name
	var projWinner = fighterCareers[player1Name].computedWinProbability > fighterCareers[player2Name].computedWinProbability ? player1Name : player2Name;

	//dream mode is 'intelligently making the stupid bet'. Because long shot bets have such high payouts they can be worth betting on 
	//if you have nothing to lose. Since you are always given 'bailout' cash if you end up with 0 or in the hole, it makes sense to 
	//bet on super long shots. If they win you get a TON of cash. If they lose you are just right back to where you started. Of course
	//that's up to the player though if they want to use that mentality so I made it optional. Also most players would only want to make stupid bets
	//if they have under a certain amount to keep from losing their fortune, and because at higher dollar values you can bet a large enough
	//percent of the total pot to still make good returns.
	if(dreamMode && saltyBucks < dreamModeDisabledAtAmount)
	{
		var winPercentDifference = fighterCareers[player1Name].computedWinProbability > fighterCareers[player2Name].computedWinProbability ? fighterCareers[player1Name].computedWinProbability - fighterCareers[player2Name].computedWinProbability : fighterCareers[player2Name].computedWinProbability - fighterCareers[player1Name].computedWinProbability;
		if(winPercentDifference > dreamModePercentThreshold)
		{
			$('#statusDiv').html('Bet on the dream!');
			 projWinner = fighterCareers[player1Name].computedWinProbability < fighterCareers[player2Name].computedWinProbability ? player1Name : player2Name;
		}
	}
	$('#statusDiv').html('Projected winner is ' + projWinner);
	return projWinner;
}

Great, now I had pretty confidently who was going to win and lose. But I was still short on data and betting manually all the time was getting to be a pain. My bot could auto bet, but not know who was fighting, or I could manually bet and have to actually enter names to do it. At this point you are probably saying, ‘well just extract the character names from somewhere, feed them into the formula and be done with it!’. I wish it was that simple. The stream of the fight is an embedded flash object and the names of the characters do not appear anywhere. The names are simply not available by any conventional means. It seriously seemed like the author went out of his way to make the names not available to prevent this kind of thing. I knew I’d have to solve that problem but, for the time being I needed to collect data. I settled on just having a stupid bot bet small amounts on someone at random so I could harvest that sweet sweet result data.

Even with that decision it wasn’t totally easy. Because it’s an embedded flash object how would I know when the betting window is open? I’ve only got about 45 seconds from when betting opens to when it closes, so whatever I do has to be reasonably quick. I then realized that the status text below the video changes to ‘Betting is Now open’ when you can bet. I simply told my bot to keep on a DOM transform onchange function to that. When that element changes evaluate the text and figure out if it says betting is now open. If so, wait about 40 seconds (so I have time to enter a manual bet if I want to) and then if no bet has been placed enter one. Using that same technique I know when the fight starts, ends, and payouts have been distributed. That ended up working out pretty well, though occasionally there seemed to be some sever delays that prevented entering a bet if I was too close to the deadline.

What my javascript bookmarklet looks like

What my javascript bookmarklet looks like

Using the same kind of trick I was able to extract the players current saltybuck total so i could deduce how to bet a small percent amount of their total, instead of just some static amount. Things were coming together well. I could just leave the bot on all night and it would bet for me. There were one or two mornings i came back and it had won me over 100K (randomly of course, it had no idea who it was betting on at this point). I build a nice little interface using jQuery UI that could be launched via my bookmarklet and if i entered the names I could get some decent odds data. I even rigged up an auto complete on the fighter names based on all the known fighters from the win/loss totals table. I added a few more fun little features, a hotkey combination to show and hide the window. I even added a sound effect ‘Oh yeah‘ if the bot wins a big amount of money (currently defined as over 10K, though I should probably make it to something like over 200% of your current total topping out at like 50K or something). When I actually paying attention and betting I was doing well, and if I walked away the bot would take over and place small bets to keep that sweet data stream coming in.

I knew that this was about as far as I could take the bot running as just a javascript thing bookmarklet thing. If I wanted more (centralized data so ELO and such didn’t have to be recalculated every fight), potentially to actually know who is fighting, I’d have to step out and really tread into unknown territory. I was going to need to somehow get a screenshot of who was fighting during the betting time. I was going to need to extract the names from the image. I’d have to feed that into some kind of optical character recognition engine (OCR). Then I’d have to take those results and make them available via a web service. I’d have to modify the bot to reach out to that webservice to trigger the reading and get the names. This couldn’t be done in the browser so I was going to need to develop some kind of server mechanism. I’d also need about 5 pots of coffee.

The Server

I decided I’d tackle what I considered to the easier part first to keep my spirits up and keep me from quitting when I reached the part which i knew would be most difficult (the OCR). The server had a fairly simple job to do in my mind. I needed to listen for a call from the client (since the client knew when the betting screen was open, it could make the callout, where as the server would have no idea because that monitoring functionally was still built into the client part. I’d have to refactor this later). When it got the request it would need to take a screenshot of the browser window which would also have to be running on the server. Ideally it would extract just name of the fighters, and save those images. It would then trigger the OCR engine to read the files. When that was done it would then read out the resulting data back to the requester (huh, now that I type that out it sound kind of hard, but regardless it wasn’t really too bad). I decided the easiest and lightest weight answer for a server would be a node js instance. I have some experience with node and it’s quick to get running so it seemed like the natural candidate.

After a bit of reading to get back up to speed on how to setup node and getting my basic hello world up and running I found a library that would allow node to execute commands on the server (yeah I know that’s dangerous, but this is all local, so whatever). I just rigged it up to listen for a specific page request, and once it got that it would run a batch file which would handle the screenshot, image processing and OCR work. Once it got word the batch file had run it would read the contents of the two text files that were hosted on the server as well that would hold the names of the current fighters. Here is the node code.

var express = require('express');
var sh = require('execSync');
var app = express.createServer() ;
var fs = require('fs');

var port = process.env.PORT || 80;
//configure static content route blah
app.configure(function(){
  app.use(express.methodOverride());
  app.use(express.bodyParser());
  app.use(express.static(__dirname + '/public'));
  app.use(express.errorHandler({
    dumpExceptions: true, 
    showStack: true
  }));
  app.use(app.router);
});

app.listen(port, function() {
  console.log('Listening on ' + port);
});

app.get('/getFighters', function(request, response){

	console.log('Request made to get fighter data');
	var result = sh.exec('cmd /C screenshot.bat');

	console.log('Command ran ' + result.stdout);
 	fs.readFile( 'public\\fighter1Name.txt', "utf-8", function (err, fighter1) {
		if (err) console.log( err );
		fs.readFile( 'public\\fighter2Name.txt', "utf-8", function (err, fighter2) {
		  if (err) console.log( err );
		  var fighters = new Object();
		  fighters.fighter1 = fighter1.trim();
		  fighters.fighter2 = fighter2.trim();

		  response.send(request.query.callback+'('+JSON.stringify(fighters)+');');
		});
	});
});

Not too bad eh? As you can see the results are wrapped using a JSONP style callback system so this can be invoked from anywhere. Once that was up and running now I had to write the batch file that actually did all the hard work.

The Bat File

The node server pretty much has a black box kind of process. It just calls some batch file and expects results. Not that it really matters, but the execute process is async and so the server didn’t know when that process had completed (ended up having to have a loop that attempts to read the contents until it succeeds, shitty I know). It has no idea of course how the bat file does what it does, and honestly neither did I when i first started building it. I knew the bat file would have to take a screenshot, extract the names of the fighters from that screenshot, and invoke the OCR engine. At this point i knew I was at least going to use Tesseract for my OCR engine, and that ImageMagick (a suite a command line tools for image processing) where likely going to be how I did the image processing. For capturing the screenshot I found a simple utility on google code called screenshot-cmd that would take a screenshot of the primary monitor. I figured then I could use imagemagick  to crop down the un-needed stuff (since the video is in the exact same place on my screen every time I could use coordinate based cropping). Then with the images cleaned up I could forward them onto Tesseract.

After a bit of messing around I managed to get the screenshot and get ImageMagick to extract just the names of the fighters from the betting screen image. Later on I had a sudden moment of clarity and realized I could remove the background from the names if I just deleted everything that wasn’t the red color for the player 1 name, and the blue color for the player 2 name (since they are always exactly the same color). Also I decided to archive the old captures so I’d have them to help train the OCR engine. The final batch script looks like this

@ECHO OFF

FOR %%I IN ("public\screens\*.png") DO (
  SET lmdate=%%~tI
  SETLOCAL EnableDelayedExpansion
  SET lmdate=!lmdate:~6,4!-!lmdate:~3,2!-!lmdate:~0,2! !lmdate:~11,2!-!lmdate:~14,2!
  MOVE "%%I" "public\screens\old\!lmdate!-%%~nxI"
  ENDLOCAL
)

::Take screenshot of primary monitor at full resolution
screenshot-cmd 0 0 1920 1080 -o public\screens\fighters.png

::ImageMagick shave off the left 478 pixels and the top 135 pixels to cleanup the image
convert -shave 478x135  public\screens\fighters.png public\screens\fighters.png

::ImageMagick remove the bottom and right borders
convert public\screens\fighters.png -gravity South  -chop  0x150  public\screens\fighters.png

::Now we have a screenshot with just the fighters. Now we have to extract the names of the fighters and put them in separate files

::Extract fighter1 name by cropping out an 800px X 40px swatch from the top of the image
convert public\screens\fighters.png -crop 800x40+60+0 public\screens\name1.png

::Remove all colors except for the red used by the font
convert public\screens\name1.png -matte ( +clone -fuzz 4600 -transparent #e3522d ) -compose DstOut -composite public\screens\name1.png

::Extract fighter1 name by cropping out an 800px X 40px swatch from the bottom of the image
convert public\screens\fighters.png -crop 800x40+200+618 public\screens\name2.png

::Remove all colors except for the red used by the font
convert public\screens\name2.png -matte ( +clone -fuzz 4600 -transparent #2798ff ) -compose DstOut -composite public\screens\name2.png

::Feed the player names into tesseract for OCR scanning.Write results to two different text files. One for each fighter
tesseract public\screens\name1.png public\fighter1Name -l salty
tesseract public\screens\name2.png public\fighter2Name -l salty

The commands took a bit of time to get just right (what with having to find just the right offsets and messing with the color removing fuzz factor). The final output is pretty damn good actually. Check this out.
name1name2

All things considered I’d say those are some damn fine extractions from a screenshot of a flash video. Now all that was left is the final part, tackling the Tesseract OCR training process to teach it about this strange font.

Tesseract OCR

Tesseract is pretty much the premier freeware OCR engine. There really isn’t anything else that competes with it. It’s hard as hell to figure out and takes a ton of time to get setup properly for new languages but I had heard when it works, it works pretty damn well. I know next to nothing of OCR, so I knew tackling this was going to be a challenge. The basic outline breaks out like this

1) Gather samples of your new font. At least a few occurrences of every possible character.

2) Create a ‘box’ file which is basically just a coordinate mapping of where each character starts and stops and what it represents. (finding a functional tool for this part took forever, because it turns out I was using a bad image that caused them all to have problems or act very slowly. Pro tip, when saving your TIF file to feed into a box editor, if using photoshop discard the layer data. It makes the file way too big and slow to use).

3) Train Tesseract using the box file

4) Generate the rest of the weird files it needs that I don’t know what do.

5) Package all the files and see if your new language works.

eng.salty.exp0The shortcut method here is create your training image with all your chars, use jTessBoxEditor to do your modifications to the box file. Then use SerakTesseractTrainer to do the training and create the files. Honestly if i had known about those two things right off the bat, my life would have been a lot easier. Over half my battle was just trying to find what tools to use and getting them to work right.

Also retraining it after I was able to remove the backgrounds from the names made it about a billion times more accurate. I would highly recommend that approach if you have the ability to. Good training data makes all the difference. Trying to train it with crummy data with backgrounds and weird shit going on makes it next to impossible. On the right you can see what my training data looked like and it ended up working out pretty well. It’s still lacking some numeric characters, but I’ll have to add those in later.

I was amazed to find it actually worked. The names were being read properly and written to the file. The node server was grabbing the contents of the file and returning it to the requesting bot. The bot took the names and fed them into the scoring system and placed bets accordingly. It was a beautiful symphony built from a total clusterfuck. I am almost sad now because I have solved my project. Sure i can make a little better, implement a database, maybe tweak the scoring engine some, but overall it’s been solved. All that’s left to do now is sit back and watch the salt roll in. Later on I did a bit of re-factoring, moving the calculation onto the server and out of the client (where it belongs). I also created an extension just for the server that would invoke the screen reading process instead of accepting the request for the normal client (since I figured I may end up distributing the code I didn’t want everyone’s clients telling my server to constantly try to re-read the screen and such). Eventually the client got dumbed down to just polling the server when it detected that bets were open until it got back the fight odds and it then could set a suggested bet amount for the player. I also ended up adding a few other features to the client like ‘dream mode’ where in if the odds against a character were so insane as to make payout on the favorite nominal but the payouts for underdog amazing, bet on the underdog in hopes of a huge payout. You could set some variables like always bet in dream mode until you reached a certain threshold. You could also bet all in mode which would automatically bet all your money until a certain threshold since payouts at lower levels of betting were always so minimal. This is what the ‘final’ version of the client ended up looking like.

saltyclient final

As a postscript to this story to gather more data I ended up offering a trade to other players. If they could provide my their betting history data and enough of it was unique (I didn’t already have the results of that fight, which I identified by timestamp I would give them access to the tool). With their betting data added onto mine I ended up having an accuracy rate of around 85% which isn’t too bad. The overall results were somewhat disappointing though because for whatever reason the SaltyBet community was really good at guessing as well and the odds would end up so heavily staked in the winners favor that usually my payouts were pretty small.

Right now the Saltybot server isn’t running and the data is probably badly out of date, but hey if you want to download the source and get it running again, knock yourself out. You can download the source here

https://drive.google.com/file/d/0B04fc3zIG4iyMURsemloR040NFE/edit?usp=sharing

I don’t remember the exact setup steps, but I believe you’ll want to drop all the server files in a directory on your machine. Spin up a node.js console and launch core.js. Open up saltybet.com and keep it fullscreen. Then on your server install the saltyBotServerExtension into chrome. That should watch for fight changes and do the OCR process and put the results into the public folder. You’ll want to setup a web server where the public folder is available for your client to get at. Then install the client extension in your machine you intend to use as your ‘betting’ machine and point it at your webserver (yeah you’ll probably have to modify the source, thankfully in chrome you can just modify the source and load the unpacked extension). That should get you pretty close. If you have questions, feel free to ask, I’ll do what I can to help. I am interested in seeing where this goes, I’m just too lazy right now to do much with it myself. If there is interest maybe I’ll try and get it running again.


Visualforce Force Download of PDF or Other Content

Hey everyone,

This next trick is one I’ve kind been keeping under my hat since it’s a nice polishing touch for some of my contest entries, but I figured I should probably share it with the world now (information must be free, etc). So we all know we can create Visualforce pages that render as PDF documents. It’s a pretty cool feature especially because business people love PDF files more than I love being a cynical ass (which is like… a lot). Though the one little annoyance is that normally when you create that PDF visualforce page the user is brought to it to view it where they then can download it. Many times they simply want to download it and attach it to an email or something, the viewing isn’t required and is generally just an extra few wasted seconds waiting for it to load so they can hit file->save as. I have found/built a nifty way to force download of the file using a combination of Apex and some tricky DOM manipulation. As an added bonus I’ll show you how to conditionally render the page as a PDF based on a URL param. Here we go!

The first thing we’ll need of course is our Visualforce page, we’ll keep it simple for this example. So here is our visualforce page

<apex:page controller="forceDownloadPDF" renderAs="{!renderAs}">
<h2>PDF Download example</h2>

<p>This is some content that could be displayed as a PDF or a regular web page depedning on the URL params. The valid URL params are as follows</p>
<table width="100%" cellpadding="5" cellspacing="5">
    <tr>
        <th>Name</th>
        <th>Type</th>
        <th>Default</th>
        <th>Required</th>
        <th>Description</th>
    </tr>
    <tr>
        <td>pdf</td>
        <td>String with a boolean value</td>
        <td>null/false</td>
        <td>false</td>
        <td>if passed in as a true the page will be rendered as a PDF. Otherwise displayed as HTML</td>
    </tr>
    <tr>
        <td>force_download</td>
        <td>String with a boolean value</td>
        <td>null/false</td>
        <td>false</td>
        <td>If true the user will be prompted to download the contents of the page. Suggested to be paired with pdf=true</td>
    </tr>
    <tr>
        <td>filename</td>
        <td>String (valid file name)</td>
        <td>'My PDF Report [todays date].pdf'</td>
        <td>false</td>
        <td>A name for the file. Only used if force_download=true</td>
    </tr>    
</table>

</apex:page>

And now our controller

public class forceDownloadPDF {

    public string renderAs{get;set;}

    public forceDownloadPDF()
    {

        //figure out if the user passed in the pdf url variable and if it is set to true.
        if(ApexPages.currentPage().getParameters().get('pdf') != null && ApexPages.currentPage().getParameters().get('pdf') == 'true') 
        {
            //if so, we are rendering this thing as a pdf. If there were other renderas options that were valid we could consider allowing the user to pass
            //in the actual renderAs type in the url, but as it stands the only options are pdf and null so no reason to allow the user to pass that in directly.
            renderAs = 'pdf';

            //figure out if we are forcing download or not.
            if(ApexPages.currentPage().getParameters().get('force_download') != null && ApexPages.currentPage().getParameters().get('force_download') == 'true') 
            {
                //setup a default file name
                string fileName = 'My PDF Report '+date.today()+'.pdf';

                //we can even get more created and allow the user to pass in a filename via the URL so it can be customized further
                if(apexPages.currentPage().getParameters().get('filename') != null)
                {
                    fileName = apexPages.currentPage().getParameters().get('filename') +'.pdf';
                }
                //here is were the magic happens. We have to set the content disposition as attachment.
                Apexpages.currentPage().getHeaders().put('content-disposition', 'attachemnt; filename='+fileName);
            }               
        }        
    }
}

As noted in the comments the real secret here is setting the content disposition use the Apex getHeaders method. Now you are saying,

‘But Kenji if I call that page from a link it still opens in a  new window it just forces the user to download the file. That’s not much better!’

Oh ye of little faith, of course I got you covered. You think I’d leave you with a half done solution like that? Hell no. Lets take this mutha to the next level. Here is what we are going to do. Using a custom button with onClick javascript we are going to create an iframe with the source set as that visualofrce page (with the force_download=true param) and inject it into the DOM. When the frame loads (which will have 0 width, and height so it’s not visible) that code still runs prompting the user to download the file. They are non the wiser that a frame got injected, all they see is a happy little download dialog prompt. So go create a custom button on an object that you want to prompt the user to download your file from. Make it a detail page button (you could do a list button to, but that’s a topic for another day). Make it onClick javascript. Then slap this code in there.

ifrm = document.createElement("IFRAME"); 
ifrm.setAttribute("src", "/apex/yourPage?pdf=true&force_download=true&filename=My Happy File"); 
ifrm.style.width = 0+"px"; 
ifrm.style.height = 0+"px"; 
document.body.appendChild(ifrm);

Of course replace the ‘yourPage’ with the name of your visualforce page. The filename of course can be changed to be details from the record, or whatever you like. Now when the user clicks that button the javascript creates an invisible iframe and injects it into the DOM. Once it loads the user is prompted to download the file. Pretty slick eh?

Hope you dig it. Catch ya next time.


Salesforce Dashboard Automatic Refresh Bookmarklet

Hey all,

Quick fun little chunk of code here for you. This code when saved as a bookmarklet (javascript saved as a bookmark which runs on the current page when clicked) will cause Salesforce dashboards to automatically refresh every X seconds, where X is a variable near the top of the code (defaults to 90 seconds). It also injects a little timer on the refresh button, and is smart enough to wait for the dashboards to refresh before it continues the next countdown. I haven’t cross browser tested it yet (built in Chrome 25) but as long as the browser supports the DOMSubtreeModified event listener you are probably fine. Just save the code as a bookmarklet, navigate to your dashboard page and click the bookmarklet. You should see a small timer show up on the refresh button. When the timer hits 0 the dashboard should refresh, and the timer will reset back to the default time and being counting down again.

javascript:(
    function() 
    {
        var refreshInterval = 90; //number of seconds between each refresh
        var counter = refreshInterval;
        var timerInterval;
        var button = document.getElementById('refreshInput');
        if(button == null)
        {
            alert('Refresh Button not found! Salesforce may have changed the buttons ID or it may not be visiable for some reason. Please make sure you are on a dashboard page with the Refresh button visible');
            return false;
        }

        document.addEventListener("DOMSubtreeModified", function() {
            if(event.target.id == "componentContentArea")
            {
                startTimer();
            }
        }, true);

        function countDown(){
            counter--;
            button.value = "Refresh ("+formatTime(counter)+")";
            if(counter == 0)
            {
                button.click();
                counter = refreshInterval;
                window.clearInterval(timerInterval);            
                button.value = "Waiting for Refresh";
            }                
        }

        function startTimer()
        {
            window.clearInterval(timerInterval);
            timerInterval = setInterval(countDown, 1000);     
        }    

        function formatTime(seconds)
        {
            var totalSec = seconds;
            hours = parseInt( totalSec / 3600 ) % 24;
            minutes = parseInt( totalSec / 60 ) % 60;
            seconds = totalSec % 60;

            result = (hours < 10 ? "0" + hours : hours) + ":" + (minutes < 10 ? "0" + minutes : minutes) + ":" + (seconds  < 10 ? "0" + seconds : seconds);            

            return result;
        }
        startTimer(); 
    }
)();

Lets Build a Tree (From Salesforce.com Data Categories)

Salesforce Data categories. If you’ve had to code around then on the Salesforce.com platform, you are probably aware of the complexity, and how much of a pain they can be. If you havn’t worked with them much, you are fortunate 😛 They are essentially a way to provide categories for any sObject in Salesforce. They are most frequently used with knowledge articles. The Apex calls, describes and schema for them is unlike anything else in the Salesforce schema. Categories are their own objects and they can be nested to infinite complexity. In short, they are complicated and take a while to really get your head around them (I still don’t know if I really do). Thankfully I’ve done a bunch of hard work and discovery so that you don’t have to. For this particular project, we are going to build a nifty tree style selector that allows a user to select any data category for a given sObject type. You can then do whatever you want with that info. Yes I know there are some built in visualforce components for handling data categories, but they aren’t super flexible and this is just a good leaning experience. In the end, you’ll have an interactive tree that might look something like this.

treeDemoWord of Warning: I had to live modify some of code I posted below to remove sensitive information that existed in the source project. I haven’t used the EXACT code below, but very very close. So please let me know if something doesn’t quite work and I’ll try to fix up the code in the post here. The idea works, it’s solid, but there might be a rough syntax error or something.

Our application is going to consist of a visualforce page that displays the tree. A component that contains the reusable tree code, a static resource that contains the javascript libraries, css file and images for the tree structure. Of course we will also have an apex class that will handle some of the heavy lifting of getting category data, and returning it to our visualforce page. We’ll use javascript/apex remoting to communicate with that Apex class. First off, lets grab the static resource and get that uploaded into your org. You can snag it here

https://www.box.com/s/04u0cd8xjtm0z84tbhid

upload that, make it public, call it jsTree. Next we’ll need our Apex class. It looks like this.

global class CaseSlaController
{
    //constructors for component and visualforce page extension
    public CaseSlaController() {}
    public CaseSlaController(ApexPages.StandardController controller) {}

    //gets category data and returns in JSON format for visualforce pages. Beware that since we end up double JSON encoding the return 
    //(once from the JSON.serialize, and another time because that's how data is returned when moved over apex remoting) you have to fix
    //the data on the client side. We have to double encode it because the built in JSON encoder breaks down when trying to serialize
    //the Schema.DescribeDataCategoryGroupStructureResult object, but the explicit call works.
    @remoteAction 
    global static string getCategoriesJson(string sObjectType)
    {
        return JSON.serialize(CaseSlaController.getCategories(sObjectType));
    }

    public static  list<Schema.DescribeDataCategoryGroupStructureResult> getCategories(string sObjectType)
    {

        //the describing of categories requires pairs of sObject type, and category name. This holds a list of those pairs.
        list<Schema.DataCategoryGroupSObjectTypePair> pairs = new list<Schema.DataCategoryGroupSObjectTypePair>();

        //list of objects to describe, for this app we only take 1 sObject type at a time, as passed into this function.
        list<string> objects = new list<string>();
        objects.add(sObjectType);

        //describe the categories for this object type (knowledgeArticleVersion)
        List<Schema.DescribeDataCategoryGroupResult> describeCategoryResult =  Schema.describeDataCategoryGroups(objects);

        //add the found categories to the list.
        for(Schema.DescribeDataCategoryGroupResult s : describeCategoryResult)
        {
            Schema.DataCategoryGroupSObjectTypePair thisPair = new Schema.DataCategoryGroupSObjectTypePair();
            thisPair.sObject = sObjectType;
            thisPair.dataCategoryGroupName = s.getName();
            pairs.add(thisPair);            
        }

        //describe the categories recursivly
        list<Schema.DescribeDataCategoryGroupStructureResult> results = Schema.describeDataCategoryGroupStructures(pairs,false);

        return results;
    }    
    private static DataCategory[] getAllCategories(DataCategory [] categories)
    {
        if(categories.isEmpty())
        {
            return new DataCategory[]{};
        } 
        else
        {
            DataCategory [] categoriesClone = categories.clone();
            DataCategory category = categoriesClone[0];
            DataCategory[] allCategories = new DataCategory[]{category};
            categoriesClone.remove(0);
            categoriesClone.addAll(category.getChildCategories());
            allCategories.addAll(getAllCategories(categoriesClone));
            return allCategories;
        }
    }
}

So there are three functions there and two constructors. The constructors are for later on when we use this thing in a component and a visualforce page, so don’t really worry about them. Next is the getCategoriesJson, that is the remote function we will call with our javascript to get the category data. It just invokes the getCategories function since that returns an object type that Salesforce can’t serialize with it’s automatic JSON serializer without blowing up (in my real app I had to use getCategories for another reason, hence why I didn’t just combine the two functions into one that always returns JSON). The last one is just a private function for spidering the data category description. Other than that, you can check out the comments to figure out a bit more about what it’s doing. In short it describes the categories for the given sObject type. It then creates dataCategoryGroupSobjectTypePairs from those categories and describes those and returns the huge complicated chunk.

Alright, so we got the back end setup, let’s actually make it do something. For that we need our component and visualforce page. First up, the component. Wrapping this picker in a component makes it easy to use on lots of different visualforce pages. It’s not required but it’s probably a better design practice.

<apex:component Controller="CaseSlaController">
    <!---- Two parameters can be passed into this component ---->
    <apex:attribute name="sObjectType" type="string" description="the sObject type to get data category tree for" />
    <apex:attribute name="callback" type="string" description="Name of javascript function to call when tree drawing is complete" />

    <!--- include the required libraries --->
    <link rel="stylesheet" href="{!URLFOR($Resource.jsTree, 'css/jquery.treeview.css')}" />
    <apex:includeScript value="{!URLFOR($Resource.jsTree, 'js/jquery.min.js')}" />
    <apex:includeScript value="{!URLFOR($Resource.jsTree, 'js/jquery.treeview.js')}" />

    <script>
        //put jQuery in no conflict mode
        j$=jQuery.noConflict();     

        //object to hold all our functions and variables, keep things organized and dont pollute the heap
        var categorySelect = new Object();

        //invokes the getCategoriesJson function on the apex controller. Returns to the callback function with the
        //fetched data
        categorySelect.getCategoryData = function(sObjectType,callback)
        {
            Visualforce.remoting.Manager.invokeAction(
                '{!$RemoteAction.CaseSlaController.getCategoriesJson}', 
                sObjectType,
                function(result, event){
                   callback(result,event);
                }, 
                {escape: true}
            );          
        }    

        //as soon as the dom has loaded lets get to work
        j$(document).ready(function() {

            //first off, find all the data category data for the given sObject type.       
            categorySelect.getCategoryData('{!sObjectType}',function(result,event)
            {
                //the json data we get back is all screwed up. Since it got JSON encoded twice quotes become the html
                //&quote; and such. So we fix the JSON and reparse it. I know its kind of hacky but I dont know of a better way

                var fixedJson = JSON.parse(categorySelect.htmlDecode(result));         

                //lets create the series of nested lists required for our tree plugin from the json data.
                var html = categorySelect.buildTreeHtml(fixedJson);                          

                //write the content into the dom
                j$('#categoryTree').html(html);              

                //apply the treeview plugin
                j$("#categoryTree").treeview({
                    persist: "location",
                    collapsed: true,
                    unique: true
                });  

                //if the string that was passed in for callback is actually representative of a function, then call it
                //and pass it the categoryTree html.
                if(typeof({!callback}) == "function")
                {
                    {!callback}(j$("#categoryTree"));                                               
                }
            });    
        });

        //function that is meant to be called recursivly to build tree structure html
        categorySelect.buildTreeHtml = function(category)
        {
            var html = '';     

            //iterate over the category data  
            j$.each(category,function(index,value)
            {
                //create list item for this item.
                html+='<li><a href="#" category="'+value.name+'" class="dataCategoryLink" title="Attach '+value.label+' SLA to Case">'+value.label+'</a>';

                //check to see if this item has any topCategories to iterate over. If so, pass them into this function again after creatining a container               
                if(value.hasOwnProperty('topCategories') && value.topCategories.length > 0)
                {
                    html += '<ul>';
                    html += categorySelect.buildTreeHtml(value.topCategories);                    
                    html +='</ul>';                 
                }   
                //check to see if this item has any childCategories to iterate over. If so, pass them into this function again after creatining a container                           
                else if(value.hasOwnProperty('childCategories')  && value.childCategories.length > 0)
                {
                    html+='<ul>';                   
                    html += categorySelect.buildTreeHtml(value.childCategories);
                    html+='</ul>';
                }
                html += '</li>';
            });
            return html;                
        }

        //fixes the double encoded JSON by replacing html entities with their actual symbol equivilents
        //ex: &quote; becomes "
        categorySelect.htmlDecode = function(value) 
        {
            if (value) 
            {
                return j$('<div />').html(value).text();
            } 
            else
            {
                return '';
            }
        }            
    </script>
    <div id="categoryTreeContainer">
        <ul id="categoryTree">

        </ul>
    </div>
</apex:component>

Now then finally we need a visualforce page to invoke our component and rig up our tree items to actually do something when you click them. We wanted to keep the component simple, just make the interactive tree cause different pages might want it to do different things. That is where that included callback function comes in handy. The visualforce page can invoke the component and specify a callback function to call once the component has finished its work so we know we can start manipulating the tree. Our page might look like this.

<apex:page sidebar="false" standardController="Case" showHeader="false" extensions="CaseSlaController">
    <c:categorySelect callback="knowledgePicker.bindTreeClicks" sObjectType="KnowledgeArticleVersion"/>

    <script>           
        var knowledgePicker = new Object();

        knowledgePicker.bindTreeClicks = function(tree)
        {
            j$('.dataCategoryLink').click(function(event,ui){
                event.preventDefault();
                alert('clicked ' + j$(this).attr('category'));
            }); 
        }                           
    </script>   
</apex:page>

We invoke the component passing it a callback function name and the type of sObject we want to make the category tree of. We then create a function with the same name as the callback. Inside that function we simple attach an onclick event handler to the tree category links that sends us an alert of which one the user clicked. Of course we could then do anything we wanted, make another remoting call, update an object, whatever.

Anyway, I hope this was helpful. I know I was a bit frustrated as the lack of sample code for dealing with categories so hopefully this helps some other developers out there who might be trying to do the same kind of thing. Till next time!

-Kenji/Dan


Javascript Console

So I just wrapped up another Cloudspokes challenge.I spent way too much time on this. Really it’s just a dumb little $300 dollar challenge, and I think I put like 10 or so hours into this, but whatever, it was fun. I probably went overboard, but I think I made something actually kind of useful. First, before I get too deep into it, check out the challenge details.

http://www.cloudspokes.com/challenges/1999

The basic idea being that they wanted a way to run javascript on a web page, similar to Firebug or the Chrome developer tools. They also suggested adding a list of the user/developer defined functions and maybe an auto complete system to make entering the code to run easier. I wanted to take it a step further and build it all as pure javascript so that it could be turned into a bookmarklet. With it all delivered as a bookmarklet that means you can inject this console on any webpage anywhere without needing access to the source and start running functions on it. You don’t need to include any additional libraries or hooks or anything. You need NO access to the source code of the webpage for this to work, which is what really makes it cool.

It does this by using javascript to dynamically inject jQuery, jQuery UI, and bootstrap (only if needed, it’s smart enough not to double include libraries), and builds the console directly by inserting it into the DOM. It uses some tricky looping and evaluation to find all exisitng functions, list them, and get the function bodies as well. It creates an input area where javascript code can be entered, and the results are displayed in a console using intelligent return type handling. Using a bit of bootstrap for autocomplete, buttons, and the collapsible side list view it creates a simple yet powerful interface. As a neat side effect I found what is probably the end all, be all way to inject scripts that may or may not be present and rely on each other. I’m hoping to make a blog post out of that technique pretty soon. It uses some neat recursion and callbacks to work it’s magic and the end result is very efficient and reliable.

Anyway, check out the video for a better understanding/demo of what it can do.

http://www.screencast.com/t/jVUo5Qsh5 <- Demo video

To try it out, make a bookmark a new bookmark, and set it’s path to this. Sorry I can’t make a bookmarklet link here for you, wordpress keeps killing the link.  It might take a little bit of time to load on very complicated pages. Also it does support pressing up and down in the input box to flip between previously entered commands.

javascript: (function(){var jsCode=document.createElement('script');jsCode.setAttribute('src','https://dl.dropbox.com/u/65280390/jsConsoleScript.js');document.body.appendChild(jsCode);}());

New Cloudspokes Entry – Salesforce Global Search

Hey everyone,

I’m just wrapping up my entry for the CloudSpokes Salesforce global search utility challenge, codename Sherlock. I’m fairly pleased with how it turned out and figured I’d post the video for all to enjoy. I can’t post the code since this is for a challenge, but feel free to ask any questions about it.

http://www.screencast.com/t/GrYnfBlJFM

Features:

  • Fast operation using JS remoting. No page reloads. Ajax style searching.
  • Search any objects. Not limited to a single object type search
  • Smart formatting adjusts search result display based on data available about the object
  • Easy customization using callbacks and jQuery binds
  • Flexible, can be modified to suit many different requirements
  • Easy to use jQuery plugin based installation
  • Efficient each search consumes only one database query
  • Reliable return type makes processing search results easy
  • CSS based design makes styling results quick and easy
  • Structured namespaced code means smaller memory footprint and less chance of collisions
  • Deployable package makes for easy install
  • Over 90% code coverage with asserts provides assurance of functionality

You can check out the full documentation and feature list here

https://docs.google.com/document/d/17-SUja_SO_Enhh8LrjzDMB7VIX6S87TmPr9yBW5z-yo/edit

I don’t know why exactly they wanted such a thing, but it was certainly fun to write!


Access Salesforce REST API with Javascript

So this has been a bit of a holy grail kind of endeavor for me. A way to access Salesforce org data using only javascript. Salesforce has a million ways to get at it’s data an interact with it, but it’s always required some kind of server side technology to create the requests and deal with authentication. Not anymore! Now before I get too far, this is just proof of concept code I’d say. It works, but I only just got it working in the last few hours and have done very minimal testing, but the concepts are solid.

The approach is create a Salesforce Apex REST service that can perform all the basic CRUD operations. You then host that REST service on a salesforce.com site. By using JSONP and callbacks you can hit the service from any javascript program, send requests and get data back. This idea was put forth by Pat Patterson via his awesome post at

http://blogs.developerforce.com/developer-relations/2012/02/quick-tip-public-restful-web-services-on-force-com-sites.html

I just took the idea an expanded upon it a bit. So first off, here is the code.

/* RestProxy
Author: Daniel Llewellyn (Kenji776@gmail.com, @Kenji776)

Description: This is a rest class that can be made available to client applications via salesforce.com
sites which will allow them to perform basic CRUD operations against the org. Before this, there was no
way I know of for pure javascript applications to 'integrate' with Salesforce. You had to deal with oAuth or 
SOAP or whatever. Stuff Javascript can't really do. I needed a way to get data in and out of a site.com hosted site
which only gives you access to javascript. So I wrote this. YOu can perform queries, updates and deletes. Very basic right
now, but could be expanded on as required. Inspired by the awesome post at

http://blogs.developerforce.com/developer-relations/2012/02/quick-tip-public-restful-web-services-on-force-com-sites.html

*/

@RestResource(urlMapping='/respondentPortal/*')
global class respondentPortal 
{
    
    
    //simple wrapper response class. Makes it so all replies via this API have the same basic structure
    //including a boolean success flag, a message, any sObjects affected and some debugging params.
    global class restResponseWrapper
    {
        string message = 'run successful';
        boolean success = true;
        list<string> params = new list<string>();       
        list<sobject> sObjects = new list<sobject>();
        string requestURI = RestContext.request.requestURI;
    }
    
    class applicationException extends Exception {}
    
    //can run queries. This one doesn't except an /objectType/Id as the other methods do. This one just takes a query as specified in the
    //?query= param of the url. It will return all the objects in the sObjects property of the returned object.
    @HttpGet
    global static void doGet() 
    {                     
        restResponseWrapper thisResponse = new restResponseWrapper();
        
        //since all requests to this service will be get, since we are operating using JSONP, we first need to figure out what they actually want to do.
        thisResponse.params = getRestParams(thisResponse.requestURI);     
        string httpVerb = thisResponse.params[1].toLowerCase();

        try
        {      
              
            if(RestContext.request.params.get('authToken') == null)
            {
                throw new applicationException('No auth token provided. Please provide one with ?authToken= in the request url');
            }
            long authToken = getAuthToken();
            long providedToken = long.valueOf(RestContext.request.params.get('authToken'));
            
            //check the auth token, provide a 20 second range where it is valid (+10 seconds, -10 seconds)
            if(providedToken == null || providedToken < authToken - 10000 || providedToken > authToken + 10000)
            {
                throw new applicationException('Invalid auth token');
            }
                     
            if(httpVerb == 'get')
            {               
                String query = RestContext.request.params.get('query');
                thisResponse.sObjects = runQuery(EncodingUtil.urlDecode(query,'UTF-8'));  
                 
            }
            else if(httpVerb == 'post')
            {
                thisResponse.sObjects.add(saveSObject(thisResponse.params[2],null,RestContext.request.params));
            }
            else if(httpVerb == 'put')
            {
                thisResponse.sObjects.add(saveSObject(thisResponse.params[2],thisResponse.params[3],RestContext.request.params));
            } 
            else if(httpVerb == 'delete')
            {
                Schema.sObjectType objectDef = Schema.getGlobalDescribe().get(thisResponse.params[2]).getDescribe().getSObjectType();     
                sObject obj = objectDef.newSobject(thisResponse.params[3]);     
                database.delete(obj);  
            }   
            else
            {
                thisResponse.success = false;
                thisResponse.message = 'method ' + httpVerb + ' not implimented';                
            }          
        } 
        catch(exception e)
        {
            thisResponse.success = false;
            thisResponse.message = e.getMessage();
        }     
        

        RestContext.response.addHeader('Content-Type', 'text/plain');
        RestContext.response.responseBody = formatResponse(thisResponse);                    

    }

    
    
    //take a query string and run it. Yeah, I know SOQL injection possible here, but whatever. This
    //whole thing is inherently insecure. It's like worrying about wet floors on the titanic.
    public static list<sobject> runQuery(string queryString)
    {
        list<sobject> objs = new list<sobject>();
        objs = database.query(queryString);
        return objs;
    }
    
    //returns rest params (things after the main url, separated by /)
    //in an easy to use list. The name of the class should be element 0, the 
    //type of object should be element 1 and the id of the object should be element 2
    global static list<string> getRestParams(string url)
    {
        list<string> returnParams = new list<string>();
        integer endOfUrl = url.indexOf('?');
        if(endOfUrl == -1)
        {
            endOfUrl = url.length();
        }
        //clean up the url, make sure we are only dealing with the section we want to. After the host name, before the ? mark.
        //i've seen in come in with the full host name, and also just the relative portion, so we gotta make sure it's exactly the same
        //from here on in, or else the array it returns could get all messed up.
        if(url.indexOf('/apexrest/') > 0)
        {   
            url = url.substring(url.indexOf('/apexrest/')+10,endOfUrl);
        }  
        else
        {
            url = url.substring(1,endOfUrl);
        }            
        list<String> URLParams = url.split('/'); 
         
        for(integer i =0; i < 4; i++)
        {
            if(i<urlParams.size())
            {
                returnParams.add(EncodingUtil.urlDecode(urlParams[i], 'UTF-8'));
            }
            else
            {
                returnParams.add(' ');
            }
        }   
        
        return returnParams;           
    }
    
    
    //so for the record, I really dislike this method. I wish I could just take the pure json data and deserialize it into
    //an sobject type defined at runtime, but as far as I know, you cannot do that (at least I couldn't find the syntax for it).
    //so instead this iterates over the keys of the sobject known from the global describe, finds matching keys in the JSON data
    //and assignes the value from the JSON to the sObject.
    public static sObject saveSObject(string objectType, string recordid, Map<String, Object> sObjectData)
    {
        //create a generic sObject to contain the update values.
        sObject updateObj;      
                
        //get the describe object for the type of object passed in (its just passed in as a string from the URL)
        Schema.sObjectType objectDef = Schema.getGlobalDescribe().get(objectType).getDescribe().getSObjectType();
        
        //find all the fields for this object type
        Map<String, Schema.SobjectField> ObjectFieldsMap = objectDef.getDescribe().fields.getMap();
        
 
        //this method can handle updates or inserts. If a record ID was passed in, 
        //cast the object as the type represented by the ID. If not, just create a
        //new object of the type found in the object describe.
        if(recordId != null)
        {
            updateObj = objectDef.newSobject(recordid);
        }
        else
        {
            updateObj = objectDef.newSobject();
        }    
        // populate the object's fields by looping over all the params in the rest request.
        for (String key : sObjectData.keySet())
        {
            // only add params if they are valid field on the object
            if (ObjectFieldsMap.containsKey(key))
            {
                //figure out the type of this field so we can cast it to the correct type
                string fieldType = ObjectFieldsMap.get(key).getDescribe().getType().name().ToLowerCase();
                
                //since I don't know how to do, or if it's even possible to do dynamic casting we need a 
                //series of if statments to handle the casting to numeric types. I think all the others should
                //be fine if left as a string. Dates might explode, not sure.
                
                
                if(fieldType == 'currency' || fieldType == 'double' || fieldType == 'percent' || fieldType == 'decimal' )
                {
                    updateObj.put(key, decimal.valueOf(string.valueOf(sObjectData.get(key)).trim())); 
                }
                else if(fieldType == 'boolean')
                {
                    updateObj.put(key, Boolean.valueOf(sObjectData.get(key))); 
                }                   
                else if(fieldType == 'date')
                {
                    updateObj.put(key, date.valueOf(sObjectData.get(key))); 
                }                
                else
                {
                    updateObj.put(key, sObjectData.get(key));
                }
            }
            else
            {
                system.debug('Invalid field: '+ key + ' for object type ' + objectType);
            }
        }
        //update/insert the object
        upsert updateObj;
        
        //return the saved object.
        return updateObj;
        
    }
        
    //take one of those wrapper objects and format it by wrapping it in a callback if needed, and serializing 
    //the result into json.    
    public static blob formatResponse(restResponseWrapper responseData)
    {
        string response;
        String callback = RestContext.request.params.get('callback');
        if(callback != null)
        {
            response = callback + '(' + JSON.serialize(responseData) + ');';
        }    
        else
        {
            response = JSON.serialize(responseData);
        }
        return blob.valueOf(response);
    }
    
    //just put some random mathemtical expression here that will only be known by the client application and 
    //this class. I know it's not super secure, but it's better than nothing. Will at least stop super novice
    //script kiddies.
    public static long getAuthToken()
    {
        return DateTime.Now().getTime() - 42131 / 3 * 8;
    }
    
    @isTest
    global static void testRespondentPortal()
    {
        // set up the request object
        System.RestContext.request = new RestRequest();
        System.RestContext.response = new RestResponse();
        
        //First lets try and create a contact.
        RestContext.request.requestURI = '/respondentPortal/post/contact';
        RestContext.request.params.put('firstname','test');
        RestContext.request.params.put('lastname','guy');
        RestContext.request.params.put('authToken',string.valueOf(getAuthToken()));
        //send the request
        respondentPortal.doGet();
        respondentPortal.restResponseWrapper createResponse= (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class); 
        
       
        //make sure the create was successful       
        system.assertEquals(true,createResponse.success,createResponse.message);
              
        //Okay, now lets try and get that contact we created
        System.RestContext.request = new RestRequest();
        string query = 'select name, id from contact where id = \''+createResponse.sObjects[0].Id+'\'';
        RestContext.request.requestURI = '/respondentPortal/get';
        RestContext.request.params.put('query',EncodingUtil.urlEncode(query,'UTF-8'));
        RestContext.request.params.put('authToken',string.valueOf(getAuthToken()));
        respondentPortal.doGet();
        
        respondentPortal.restResponseWrapper queryResponse = (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class);        

        system.assertEquals(true,queryResponse.success,queryResponse.message);
        system.assertEquals(1,queryResponse.sObjects.size());
        
        //cool, now lets do an update on that contact
        System.RestContext.request = new RestRequest();
        RestContext.request.requestURI = '/respondentPortal/put/contact/'+createResponse.sObjects[0].Id;
        RestContext.request.params.put('firstname','frank');
        RestContext.request.params.put('authToken',string.valueOf(getAuthToken()));
        respondentPortal.doGet();
        respondentPortal.restResponseWrapper updateResponse= (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class);

        system.assertEquals(true,updateResponse.success,updateResponse.message);
        system.assertEquals(1,updateResponse.sObjects.size());   
        system.assertEquals('frank',updateResponse.sObjects[0].get('firstname'));
        
        //and finally we'll delete them 
        System.RestContext.request = new RestRequest();
        RestContext.request.requestURI = '/respondentPortal/delete/contact/'+createResponse.sObjects[0].Id;    
        RestContext.request.params.put('authToken',string.valueOf(getAuthToken()));
        respondentPortal.doGet();
        respondentPortal.restResponseWrapper deleteResponse= (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class); 
        system.assertEquals(true,deleteResponse.success,deleteResponse.message);  
        
        //now try to make a call without an auth token. it should bomb 
        System.RestContext.request = new RestRequest();
        RestContext.request.requestURI = '/respondentPortal/get';
        RestContext.request.params.put('query',EncodingUtil.urlEncode(query,'UTF-8'));
        respondentPortal.doGet();
        respondentPortal.restResponseWrapper noAuthTokenResponse= (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class);

        system.assertEquals(false,noAuthTokenResponse.success,noAuthTokenResponse.message);   


        //now try to make a call with a bad auth token. it should bomb 
        System.RestContext.request = new RestRequest();
        RestContext.request.requestURI = '/respondentPortal/get';
        RestContext.request.params.put('query',EncodingUtil.urlEncode(query,'UTF-8'));
        RestContext.request.params.put('authToken','12345678');
        respondentPortal.doGet();
        respondentPortal.restResponseWrapper badAuthTokenResponse= (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class);

        system.assertEquals(false,badAuthTokenResponse.success,badAuthTokenResponse.message); 
                
        //try to make a call to a non existing http verb   
        System.RestContext.request = new RestRequest();
        RestContext.request.requestURI = '/respondentPortal/herpDerp';
        RestContext.request.params.put('query',EncodingUtil.urlEncode(query,'UTF-8'));
        RestContext.request.params.put('authToken',string.valueOf(getAuthToken()));
        respondentPortal.doGet();
        respondentPortal.restResponseWrapper badVerbResponse= (restResponseWrapper) JSON.deserialize(RestContext.response.responseBody.toString(),restResponseWrapper.class);

        system.assertEquals(false,badVerbResponse.success,badVerbResponse.message); 
        
        blob jsonResponse = formatResponse(badVerbResponse);   
      
             
    }
}

And here is a basic test page to show you how to invoke it with javascript and handle the results, along with how to generate a matching auth token.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js"></script>
<script src="core.js"></script>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Respondent Portal JS Library Test</title>

<script>

    // JavaScript Document
    var portal = new Object();
    
    portal.restEndPoint = 'https://yourendpoint.testbed.cs7.force.com/yoursite/services/apexrest/respondentPortal/';
    
    portal.runQuery = function(queryString,callback)
    {
        var jqxhr = $.getJSON(portal.restEndPoint+'get/?callback=?', {authToken:portal.generateAuthToken(),query: encodeURI(queryString)},  function(response) {
          console.log('queryRan');
          callback(response);
        })        
    }
    
    portal.generateAuthToken = function generateAuthToken()
    {
        return Math.round(new Date().getTime() - 42131 / 3 * 8);
    }
    
    portal.runQuery('select name, id from contact',function(result){
        console.log('Callback ran');
        console.log(result);
    });
</script>
</head>

<body>
</body>
</html>

The basic structure of a call being

https://yourorg.cs7.force.com/yoursite/services/apexrest/respondentPortal/%5BhttpVerb%5D/%5BobjectType%5D/%5Bid%5D?sObjectfield=value&othersObjectfield=otherValue
So to perform an update you might call.

https://yourorg.cs7.force.com/yoursite/services/apexrest/respondentPortal/put/contact/003M000000DSvWT?email=Kenji776@gmail.com

To do a create it would be

https://yourorg.cs7.force.com/yoursite/services/apexrest/respondentPortal/post/contact?firstname=Dan&lastname=Llewellyn

A delete would be

https://yourorg.cs7.force.com/yoursite/services/apexrest/respondentPortal/delete/contact/003M000000DSvWT

and finally a query.

https://yourorg.cs7.force.com/yoursite/services/apexrest/respondentPortal/get/?query=select id, name from contact

Also, a nice feature of the update and create, you don’t have to worry about only passing in valid sObject fields. It will use everything it can, and discard the rest. So if you arn’t sure if a field exists, or if you have rights to it, no worries. The program won’t exlode. It might not perfectly support dates or relationships yet though.

Any of those calls can accept a ?callback=functionName param which will cause the service to return the data in a callback which can be used for doing JSONP style transactions. EX

<script>
$.getJSON("https://yourorg.cs7.force.com/yoursite/services/apexrest/respondentPortal/get/?query=select id, name from contact&callback=?",
 function(data) {
    $('#data').html(data);
 });
</script>

Of course this service is a big ass security risk, so you’d likely want to at least add some kind of password to your requests, or something. Maybe a key that cycles daily that both your class and your javascript can calculate? I dunno. At least now you can get at your org data using javascript, which to me, is a big win.

EDIT: The new version of the code posted above includes a super basic authToken system. All it requires is that the client application and the REST service have a pre agreed upon method for generating a one time use key per request. I just used epoch time with some totally random changes to it, and then added a 10 second buffer. It’s pretty lame I know, but it’s better than nothing at least. Try and protect the formula you use to generate the key.

There ya go. Still a little rough around the edges, but should be enough to get you started.


Node.js, Socket.io, and Force.com canvas and you

So I got back from Dreamforce a week ago, and my head hasn’t stopped spinning. So many cool technologies and possibilities, I’ve been coding all week playing with all this new stuff (canvas, node.js, socket.io, nforce, heroku, streaming api, etc). Cloudspokes conveniently also had a challenge asking us to write a force.com canvas application that used a node.js back end. I wanted to take this chance to see if I could put what I had learned into practice. Turns out, it’s not too hard once you actually get your dev environment set up, and get things actually uploading and all the ‘paper work’ done. I wanted to leverage the Salesforce.com streaming API, Node.Js, and Socket.io to build a real time data streaming app. I also wanted to use Force.com canvas to get rid of having to worry about authentication (honestly the best part about canvas, by a wide margin). You can see the end result here:

http://www.screencast.com/t/Qfen94pl (Note the animations don’t show too well in the video due to framerate issues with the video capture software).

You can also grab my source project from here

Demo Source

Getting Set Up

First off, huge word of warning. All this process is just what I’ve found to work from trial and error and reading a ton of shit. I have no idea if this is the recommended process or even a good way to do things. It did/does however work. This was my first ever node.js application, as well as my first time using canvas and only my second time using Heroku. So ya know, definitely not pro level here but it’s at least functional. Also the actual idea for this application was inspired by Kevin O’Hara (@kevinohara80) and Shoby Abdi (@shobyabdi) from their streaming API session in dreamforce. They are also the authors of the kick ass nforce library, without which this app would not be possible, so thanks guys!

So how can you get started doing the same? Well first of course get yourself setup with Heroku. Heroku is where we are going to store our project, it’s a free hosting platform where you can host node.js, python and java applications. So if you don’t have a Heroku account, go get one, it’s free.

You’ll also want to download the Heroku toolbelt. This is going to get us our tools for dealing with Heroku apps (heroku client), as well as testing our stuff locally (foreman), and working with git. You can grab it here. https://toolbelt.heroku.com/. On that page it walks you through creating your project. For a more in depth guide, check out https://devcenter.heroku.com/articles/nodejs. Get a node.js project created and move onto the next step.

So now I assume you have a basic node.js project on Heroku. Now to actually make it do stuff, we’ll need to install some libraries using NPM. Open a command prompt and navigate to your local project folder. Install express (npm install express), socket.io (npm install socket.io) and nforce (npm install nforce). This should add all the required libraries to your project and modify the package.json file that tells Heroku the shit it needs to include.

You’ll also need a winter 13 enabled salesforce org to start building canvas apps. So go sign up for one here (https://www.salesforce.com/form/signup/prerelease-winter13.jsp) and get signed up. Depending when you are reading this you may not need a prerelease org, winter 13 may just be standard issue. Whatever the case, you need at least winter 13 to create canvas apps. As soon as you get your org, you’ll also probably want to create a namespace. Only orgs with namespaces and publish canvas apps, which you may want to do later. Also having a namespace is just a good idea, so navigate to the setup->develop->packages section, and register one for yourself.

In your org, you’ll need to configure your push topic. This is the query that will provide the live streaming data to your application.Open a console or execute anonymous window, and run this:

PushTopic pushTopic = new PushTopic();
pushTopic.ApiVersion = 23.0;
pushTopic.Name = 'NewContacts';
pushtopic.Query = 'Select firstname, lastname, email, id from contact;
insert pushTopic;
System.debug('Created new PushTopic: '+ pushTopic.Id); 

This will create a live streaming push topic in your org of all the new incoming contacts. You could change the query to whatever you want of course but for the purpose of this example, lets keep it simple.

Next, you’ll want to configure your canvas application. In your org, go to setup->create->apps. There should be a section called connected apps. Create a new one. Give it all the information for your Heroku hosted application. Permissions and callbacks here are a bit un-needed (since canvas will be taking care of the auth for us via a signed request) but should be set properly anyway. The callback url can be just the url of your application on Heroku. Remember only https is accepted here, but that’s okay because Heroku supports https without you having to do anything. Pretty sweet. Set your canvas app url to the url of your Heroku application and set access method to post signed request. That means when your app is called by canvas, it’s going to be as a post request, and in the post body is going to be a encoded signed request that contains an oAuth key we can use to make calls on behalf of the user. Save your canvas application.

The actual code (there isn’t much of it)
So we have everything configured now, but no real code. Our app exists, but it doesn’t do shit. Lets make it do something cool. Open up node server file (it’s probably called like web.js, or maybe app.js if you followed the guide above. It’s going to be whatever file is specified in your Procfile in your project). Paste this code. You’ll need to modify the clientId and clientSecret values from your canvas application. They are the consumer key and consumer secret respectively. I honestly don’t know if you’d need to provide your client secret here since the app is already getting passed a valid oAuth token, but whatever, it can’t hurt.

var express = require('express');
var nforce = require('nforce');
var app = express.createServer() , io = require('socket.io').listen(app);

var port = process.env.PORT || 3000;
//configure static content route blah
app.configure(function(){
  app.use(express.methodOverride());
  app.use(express.bodyParser());
  app.use(express.static(__dirname + '/public'));
  app.use(express.errorHandler({
    dumpExceptions: true, 
    showStack: true
  }));
  app.use(app.router);
});

app.listen(port, function() {
  console.log('Listening on ' + port);
});

io.configure(function () { 
  io.set("transports", ["xhr-polling"]); 
  io.set("polling duration", 10); 
});

var oauth;

var org = nforce.createConnection({
      clientId: 'YOUR CAVANAS APPLICATION CONSUMER KEY',
      clientSecret: 'YOUR CANVAS APPLICATION CLIENT SECRET',
      redirectUri: 'http://localhost:' + port + '/oauth/_callback',
      apiVersion: 'v24.0',  // optional, defaults to v24.0
      environment: 'production'  // optional, sandbox or production, production default
});

//on post to the base url of our application
app.post('/', function(request, response){
    //get at the signed_request field in the post body
    var reqBody = request.body.signed_request;   

    //split the request body at any encountered period (the data has two sections, separated by a .)
    var requestSegments = reqBody.split('.'); 

    //the second part of the request segment is base64 encoded json. So decode it, and parse it to JSON
    //to get a javascript object with all the oAuth and user info we need. It actually contains a lot of 
    //data so feel free to do a console.log here and check out what's in it. Remember console.log statments 
    //in node run server side, so you'll need to check the server logs to see it, most likely using the eclipse plugin.   
    var requestContext = JSON.parse(new Buffer(requestSegments[1], 'base64').toString('ascii'));
    
    //create an object with the passed in oAuth data for nForce to use later to subscribe to the push topic
    oauth = new Object();
    oauth.access_token = requestContext.oauthToken;
    oauth.instance_url = requestContext.instanceUrl;
    
    //send the index file down to the client
    response.sendfile('index.html');

});


//when a new socket.io connection gets established
io.sockets.on('connection', function (socket) {
      
    try
    {
      //create connection to the NewContacts push topic.
      var str = org.stream('NewContacts', oauth);
    
      //on connection, log it.
      str.on('connect', function(){
        console.log('connected to pushtopic');
      });
    
      str.on('error', function(error) {
         socket.emit(error);
      });
    
      //as soon as our query has new data, emit it to any connected client using socket.emit.
      str.on('data', function(data) {
         socket.emit('news', data);
      });
    }
    catch(ex)
    {
        console.log(ex);
    }
    
});

Now you’ll also need the index.html file that the server will send to the client when it connects (as specified by the response.sendfile(‘index.html’); line). Create a file called index.html, and put this in there.

<!DOCTYPE html>
<html>
    <head>
        <!-- - does a commit work remotely as well? -->
        <title>New Contacts</title>
        <meta name="apple-mobile-web-app-capable" content="yes" />
        <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
        
        <link href='http://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
        
        <link rel="stylesheet" href="/reveal/css/main.css">
        <link rel="stylesheet" href="/reveal/css/theme/default.css" id="theme">    
        
        <script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
    
        <script src="/socket.io/socket.io.js"></script>
    
        <script>
          $.noConflict();    
          var socket = io.connect();

          socket.on('news', function (data) {
            jQuery('.slides').append('<section><h2><a href="https://na2.salesforce.com/'+data.sobject.Id+'">'+data.sobject.FirstName+' '+data.sobject.LastName+'</a></h2><br>'+data.sobject.Email+'<br/></section>');
            
            Reveal.navigateNext();
          });



        </script>    
    </head>
    <body>

        <div class="state-background"></div>
        
        <div class="reveal">
            <div class="slides"> 
                <section>New Contacts Live Feed</section>
            </div>

            <!-- The navigational controls UI -->
            <aside class="controls">
                <a class="left" href="#">◄</a>
                <a class="right" href="#">►</a>
                <a class="up" href="#">▲</a>
                <a class="down" href="#">▼</a>
            </aside>

            <!-- Presentation progress bar -->
            <div class="progress"><span></span></div>                
        </div>


            
            <script src="/reveal/lib/js/head.min.js"></script>
            <script src="/reveal/js/reveal.min.js"></script>    
            <script>
                
                // Full list of configuration options available here:
                // https://github.com/hakimel/reveal.js#configuration
                Reveal.initialize({
                    controls: true,
                    progress: true,
                    history: true,
                    mouseWheel: true,
                    rollingLinks: true,
                    overview: true,
                    keyboard: true,
                    theme: Reveal.getQueryHash().theme || 'default', // available themes are in /css/theme
                    transition: Reveal.getQueryHash().transition || 'cube', // default/cube/page/concave/linear(2d)
    
                });

        
            
                    
            </script>            
    </body>
</html>

We are also going to need the CSS reveal framework to create the awesome slideshow. Grab it https://github.com/hakimel/reveal.js. In your Heroku project create a folder called public. In there create a folder called reveal. In that folder dump the css, js, lib and plugin folders from reveal. So it should be like root_folder->public->reveal->js->reveal.js for example. There is probably a more ‘git’ way to include the reveal library, but I don’t know what it is. So for now, moving folders around should work.

Now use git to push this all up to Heroku. I’d really highly recommend using the Heroku plugin for eclipse to make life easier. There is an install guide for it here https://devcenter.heroku.com/articles/getting-started-with-heroku-eclipse. However you do it, either from eclipse or command line, you gotta push your project up to Heroku. If you are using command line, I think it’s something like “git add .” then “git commit” then “git push heroku master” or something like that. Just use the damn eclipse plugin honestly (right click on your project and click team->commit, then right click on the root of your project and click team->push upstream).

If your app pushes successfully and doesn’t crash, it should run when called from Canvas now. Canvas will call your Heroku application using a post request. The post request contains the signed request data including an oAuth token. We use that oAuth token and store it in our node.js app for making subsequent api calls. Node.js returns the index.html file to the client. The client uses socket.io to connect back to the server. The server has a handler that says upon a new socket.io connection, create a connection to the push topic newContacts in salesforce, using the oAuth token we got before. When a new event comes from that connection, use Socket.IO to push it down to the client. The client handler says when a new socket.io event happens, create a new slide, and change to that slide. That’s it! It’s a ton of moving parts and about a million integration, but very little actual code.

Enjoy and have fun!


Experimental: Progressively enhance links for ajax with bookmarking and back button support!

Hey All,

So this is a bit of an ‘in progress/experimental’ kind of post. I think many of us have run into this dilemma. We want to make an awesome ajax powered website with fast loading pages and neat interface. Problem is ajax content sucks for search engine optimization, can be tricky to get bookmarking to work, and of course the back and forward buttons cause problems too. All this seems like it might make ajax a bad idea for navigation, but it’s just too cool to give up. So how can we resolve all these issues and use the awesome ajax navigation we want to? We address each challenge one by one (or just skip the bottom and copy and paste my full code example. Whatever works for you).

Progressive Enhancement (AKA dealing with shitty browsers or search engine)
My first attempts with Ajax navigation where simply to replace the href attributes of my links with javascript function calls. This really is the most straight forward approach, but the most flawed as well. Anyone who doesn’t have javascript support, or search engines trying to crawl your site won’t be able to follow your links. Your site won’t get indexed, and you’ll be abandoned by the search engine gods. Also, if you are using any kind of CMS (such as site.com from Salesforce as I am) the links created will be standard links. Your website people would have to call on you all the time to change their links, if it’s even possible! The answer to all of these problems is progressive enhancement. Use javascript to transform your regular links into javascript ajax links. This ensures that those people/bots not using javascript can still browse your site in the traditional manner. So for this example, my regular link might look like

<a href='contactUs' title='contactUs' class='ajaxLink'>Contact Us</a>

Pretty simple standard link. However I’ve added a class to it that will easily allow me to select it later with some jQuery magic later on. Now we need some javascript to turn that link from a plain jane href into a sexy ajax link. Something like this oughtta do the trick.

    $('.ajaxLink').each(function(index){
        $(this).attr('href', $(this).attr('href').replace('/','#'));
    });
    
    $('.ajaxLink').bind('click', function() {
            loadLink($(this).attr('href').replace('#','/')) 
    });  

(Yes I know it’s a bit sloppy with the replace statement. With the CMS we are using it’s a flat hierarchy, so I don’t need to worry about multiple slashes in the URL. Also I’m purposely leaving it a little less than maximally elegant to increase readability for my readers. I know I could consolidate the two loops.)

What’s happening here is that we are using jQuery to modify every link that has the ajaxLink class to replace the initial slash (which all my links will have) with a # sign. That # is magical. It’s called a hash mark and it’s original use was to make bookmarkable locations in your document. You click the link with the # you go to that location in the same document designated by the #. The # and it’s content is never sent to the server, it exists entirely client side (not that that really matters right now though). So when a user clicks it, their URL changes in their browser now, but it doesn’t cause a page reload. You hear that? Let me repeat. IT CHANGES THE URL, BUT DOES NOT CAUSE A PAGE RELOAD. That’s important. The second part creates a bound function that when you click our link it’s going to call a function called loadLink which expects to receive a valid URL (relative to the current document) so we need to flip the hash back to being a slash (I guess we could probably leave the slash out and just remove the # but whatever). We now have a system that will leave function links for those without javascript and transform them into ajax links for those who do. Sweet.

Bookmarking and unique page urls (The magic of the hash)
You may ask why even bother with the hash at all if we are just flipping it back to a slash. The reason is since it causes the URL to change, the user now has something they can bookmark. It also gives each page a unique URL with which to access it. As the user is navigating around your site, if they end up at a some buried 3 level deep page but the URL hasn’t changed at all they have no idea where they are really at. They don’t have a bookmarkable link, or one they can share with their friends. Of course the each page does have it’s own unique URL (thats how search engines and non javascript browsers will get to them) but your ajax enabled users won’t know that without the hash. With our function we wrote above, regular links now act as javascript and since they links have a # in front of them, the browser treats them as anchors. The URL changes when the link is clicked, but no page reload is preformed. This is a good thing. But wait, just because the hash is in the URL that doesn’t mean it’s really doing anything yet. If someone bookmarks your page with the hash in it, but you don’t have any handler for it nothing really happens. When our page loads we need to check and see if there is a hash in the URL. If so load the page indicated by it, if not then just load your default page. That functionality looks a bit like this.

$(document).ready(function()
{
          var hashMark = getHash();
          if(hashMark.length > 0)
          {
              loadLink (hashMark);  
          }          
    });    
});

function getHash() 
{
          var hash = window.location.hash;
          return hash.substring(1); // remove #
}

Pretty simple. All you are doing is saying when the document loads, see if there is a hash mark. If so, load the link indicated by it (by passing the hash mark content to the loadLink function). This works great. Now you can have bookmarkable links that actually work. But the back button is still broken….

Dealing with the back button

Man, I love jQuery. Every time I have some crazy issue to deal with, it’s got my back. Like if me and jQuery were in a bar and some big biker dude was trying to hassle me, jQuery would like tap on his shoulder, the biker would turn around and jQuery would just knock like all his teeth out with one right hook. I’d then jQuery a drink and we’d talk about how much mootools sucks (just kidding, I don’t know anything about it). Anyway where I’m going with this is that something that could be really hard to do jQuery makes really easy for us. What we need to do to get the back button to work is to detect when the hash in the URL changes. When a user clicks back or forward using your links the only thing that is going to change is that hash mark content. Nothing gets sent to the server. There is no get/post request going on here. Many hackey approaches are out there from disabled the back button to overriding it’s behavior. Thankfully we aren’t savages. We have an elegant solution. It looks like this.

          $(window).bind('hashchange', function() {
              var hashMark = getHash();
              if(hashMark.length > 0)
              {
                 loadLink (hashMark);  
              } 
          });

Just that easy. A topic that has stumped top web developers for years all wrapped up 7 lines. This just says bind a function to the hashchange function. When it changes, get the hash and pass it to the loadLink function. Boom. Done.

Loading the content (The grand finally)

So we are just about home now. We have progressive enchantments, bookmarking/link sharing ability, and even back/forward functionality. But now we need to actually load the content. Here is one last issue to deal with. Since all of your pages contain all the content/styles/scripts needed to be seen on their own (again for the non ajax users) if you try and load the entire page when you click the link you are going to end up with recursive pages nesting inside each other, duplicate script errors all kinds of crazy shit. So you need to leave the full pages intact for your non ajax users, but still be able to extract just the content you want from it to display on your page. Here we are going to use a little bit of jQuery’s find magic to extract just the content we want.

function loadLink(link)
{
     try
     {
         $("#leftText").html('');
         $('#loader').show();
         
         $.get(link, function(html) {
               $('#loader').hide();
               $("#leftText").html($(html).find("#leftText").html());
         });
     }
    catch(ex)
    {
        console.log(ex);
        $('#loader').hide();
    }
}

So here is whats happening here; the function loadLink expects to receive a valid URL fragment to load. It’s going to blank out my content area (which is called leftText) and then show an ajax loading spinner. jQuery is going to create a get request to get the link and with the result it’s going to extract the content from it’s leftText div, and insert it into this pages leftText div. Since every page is structured basically the same, it works pretty slick. That’s it. You’re done! Of course these scripts need some refining, error handling, edge case handling but I’ll leave that to the reader. The hard shit is done, what do you want me to do your whole job for you? XD Below is the entire script. Enjoy!

$(document).ready(function () {

        markupLinks();

        $(window).bind('hashchange', function () {
            var hashMark = getHash();
            if (hashMark.length > 0) {
                loadLink(hashMark);
            }
        });

        var hashMark = getHash();
        if (hashMark.length > 0) {
            loadLink(hashMark);
        }
});



function markupLinks() {
    $('.ajaxLink').each(function (index) {
        $(this).attr('href', $(this).attr('href').replace('/', '#'));
    });

    $('.ajaxLink').bind('click', function () {
        loadLink($(this).attr('href').replace('#', '/'))
    });
}

function loadLink(link) {
    try {
        $("#leftText").html('');
        $('#loader').show();

        $.get(link, function (html) {
            $('#loader').hide();
            $("#leftText").html($(html).find("#leftText").html());
        });
    } catch (ex) {
        console.log(ex);
        $('#loader').hide();
    }
}

function getHash() {
    var hash = window.location.hash;
    return hash.substring(1); // remove #
}

Apex Captcha with Javascript Remoting and jQuery

So at one time or another, we’ll likely all have to create a public facing form to collect data. We will also find about 2 seconds afterwards that it is getting spammed to hell. To stop the flood of crap, we have reCaptcha. An awesome little utility that will prevent bots from submitting forms. You already know what captcha is though, that’s probably how you found this post, by googling for apex and captcha. First off, there is already an awesome post on how to do with by Ron Hess (Here), but his approach is a bit complicated, and visualforce heavy. Of course being kind of anti visualforce, and the complexities of properties and all that, I made my own little approach. So here we go.

This is assuming you already signed up with reCaptcha. You can go here and sign up for recaptcha (yes you can just enter force.com as the domain)
After that, course add an entry for google to your remote sites in the admin setup under security. Disable protocol security.
Then create your visualforce page, and apex class. I called my class utilities, since this is kind of a re-usable function and I wanted to keep it generic.

Now put this crap in your controller. Also, your controller needs to be global (to use javascript/apex remoting)

@RemoteAction
    global static boolean validCaptcha(string challenge, string response)
    {
      boolean correctResponse = false;
      string secret = 'your recaptcha secret key here. Maybe make this into a custom setting?';
      string publicKey = 'your recaptcha public key here. Maybe make this into a custom setting?';
      string baseUrl = 'http://www.google.com/recaptcha/api/verify'; 

      string body ='privatekey='+ secret +  '&remoteip=' + remoteHost() + '&challenge=' + challenge + '&response=' + response + '&error=incorrect-captcha-sol';
      
      HttpRequest req = new HttpRequest();   
      req.setEndpoint( baseUrl );
      req.setMethod('POST');
      req.setBody ( body);
      try 
      {
        Http http = new Http();
        HttpResponse captchaResponse = http.send(req);
        System.debug('response: '+ captchaResponse);
        System.debug('body: '+ captchaResponse.getBody());
        if ( captchaResponse != null ) 
        {  
            correctResponse = ( captchaResponse.getBody().contains('true') );
        }          
       
      } 
      catch( System.Exception e) 
      {
         System.debug('ERROR: '+ e);
      }                             
      return correctResponse;
    }

    global static string remoteHost() 
    { 
        string ret = '127.0.0.1';
        // also could use x-original-remote-host 
        try
        {
            map<string , string> hdrs = ApexPages.currentPage().getHeaders();
            if ( hdrs.get('x-original-remote-addr') != null)
            {
                ret =  hdrs.get('x-original-remote-addr');
            }
            else if ( hdrs.get('X-Salesforce-SIP') != null)
            {   
                ret =  hdrs.get('X-Salesforce-SIP');
            }
        }
        catch(exception e)
        {
        
        }
        return ret;
    }

Ok, great, now your controller is ready. You just need to pass the right info and it will tell you if it’s right or wrong. Lets get a visualforce page set up to do that.

<apex:page controller="utilities" standardStylesheets="false" sidebar="false"  >

<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" />


<script>
$(function() {

    $( "#validateButton" ).click(function(){
        
        validCaptcha(function(valid){
            if(valid)
            {
                $('#validationResultDiv').html('Valid Captcha!');
                
                //Do whatever here now that we know the captcha is good.
            }
            else
            {
                $('#validationResultDiv').html('Invalid Captcha Entered');
            }
           
        });           
    });
});

function validCaptcha(callback)
{
    var challenge = document.getElementById('recaptcha_challenge_field').value;
    var response = document.getElementById('recaptcha_response_field').value;

    utilities.validCaptcha(challenge,response, function(result, event)
    {
        if(event.status)
        {
           callback(result);
        }
    }, {escape:true});
}

</script>

<div id="captchaEnter" title="Form Submission Validation">
    <center>
    <script type="text/javascript" src="https://www.google.com/recaptcha/api/challenge?k=YOUR PUBLIC KEY GOES HERE DONT FORGET IT"></script>
    <noscript>
       https://www.google.com/recaptcha/api/noscript?k=YOUR_PUBLIC_KEY
     </noscript>  
     <div id="validationResultDiv"></div>   
     <button id="validateButton" class="inline">Submit</button>
       
     </center>
</div>


</apex:page>

Boom! Just that easy. Hook up an event handler to the submit button that runs the validCaptcha function. It will get the proper values, and send them to the apex class, which sends them to reCaptcha to verify. Once an answer comes back, it is passed into the callback function, which the can run whatever action you require. Don’t forget to replace the place holder public key in the script line above. Have fun!



Ask Kenji: Cross domain ajax requests?

I was just killing some time, chilling out after karate, and this message popped in my inbox.

Hi Kenji,

I have read some articles about Salesforce in your bolg. So I have a question want to ask you, I think maybe you can give me some advices.

I want to ues jquery.ajax method to invoke the apex class.
These jquery codes are written in a HTML page not a visualforce page. I have some method to get the access_token based on the OAuth 2.0.
Then I follow your article (https://iwritecrappycode.wordpress.com/2011/07/08/salesforce-rest-api-musings/) create a apex to listen my request. I use curl to test this class and successful.
So, I think I can use jquery.ajax do the same thing.

I post the same question on force.com boards, you can see the detail at there.(http://boards.developerforce.com/t5/Apex-Code-Development/Ues-jquery-ajax-to-invoke-apex-class/td-p/394713)

Do you have experience on this?

Thank you!

A valid question. I feel like I might have touched on it before, but hey no harm in writing about it again. It’s a common situation, and one with probably more than one solution. Below is my approach. Take it or leave it.

First off, as far as I know you can’t invoke a rest resource with pure javascript. The cross domain security issue just doesn’t allow for it. The only way to do cross domain ajax stuff is by tricking the browser to loading the remote resource as if it was a script resource, since those can be loaded from anywhere. This technique in jQuery is called jsonP. The problem with this is that you cannot set headers, include authorizations, or anything else that you do with a more complex http request. It’s a simple GET to the url, and that’s is. REST resources typically require an authorization header to be set, and need to support POST, PATCH, PUT, along with just GET. So most REST resources, including the ones you can make in Salesforce can’t be access directly via javascript. If someone can prove me wrong, I love you.

So what are we do to? The method that follows is what I’ve been doing when I need a pure javascript solution. It’s not the most elegant, but it works. Here is what you have to do (this method will also get around having to use REST services, or oAuth). First, setup an visualforce page with your apex class as the controller. Wrap the return data in the callback function provided via the jQuery get request, and print the results out. Host the visualforce page on a publicly accessible salseforce site (don’t forget to set permissions on the page and class to allow the public profile user to get access) jQuery will get the response, pass the data to the inline handler function and you can process the results as you need.

<apex:page showHeader="false" sidebar="false" standardStylesheets="false" contentType="application/x-JavaScript; charset=utf-8" controller="jsonp_controller">{!returnFunction}</apex:page>

Your controller will look something like this

public class jsonp_controller
{
    public string returnFunction{get;set;}
    
    public pageReference getReturnFunction()
    {
        //get the parameters from the get/post request and stash em in a map
        map<string,string> params  = ApexPages.currentPage().getParameters();
        
        //set your data to return here
        string returnData = 'blah';
        
        if(params.containsKey('callback'))
        {
            returnFunction = params.get('callback') + '(' + returnData + ');';
        }
        
        return null;
    }
    
    @isTest
    public static void test_jsonp_controller()
    {
        jsonp_controller controller = new jsonp_controller();
        system.assertEquals(controller.returnFunction,'blah');
        
        
    }
}

And finally your page that actually makes the request would look like this

<html>
<head>
    <title>Cross domain ajax to request to salesforce</title>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.5.2/jquery.min.js"></script>
    <script>
    var url = "http://yourSite.force.com/jsonp_getData";

    function loadData()
    {
        jQuery.getJSON(url+'?callback=?',function(data)
        {
            jQuery('#results').html(data);
        });

    }

    $(document).ready(function() {
        loadData()
    });
    
    </script>
</head>
    <body>
    <div id="results">
    Your remotly fetched data will get loaded here
        </div>
    </body>
</html>

Remember, your visualforce page that serves the content must be publicly available, that means hosting it on a force.com site. Please note I wasn’t able to actually test the above code, because my org is freaking out on me right now (seriously, it’s doing some weird stuff), but it should be pretty close to accurate. Anyway, I hope this helps some people out there.

PS: I knew this topic seemed familiar. It’s because i wrote about it before!
Salesforce SOQL Query to JSON


Reliably injecting jQuery and jQuery UI with callback!

Hey all,
So this is kind of a cool thing. Sometimes you end up needing to inject jQuery in a page (like with advanced custom buttons in Salesforce) or in other cirumstances where you can write scripts, but you don’t have direct access to the source doc. Some of these times you want to include jQuery, along with jQuery UI and it’s CSS. Most of us know you can use the head.addScript function of javascript to inject the code, but how do you know when it’s loaded? How do you make sure you only load the UI library after the core library has loaded? Well worry no more, as I have an awesome javascript function here to reliably inject jQuery and the UI and then call a function of your choosing. Here ya go!

function loadJQuery(callback)
{
    try
    {
        if(typeof jQuery == "undefined" || typeof jQuery.ui == "undefined")
        {
            var maxLoadAttempts = 10;
            var jQueryLoadAttempts = 0;
            //We want to use jQuery as well as the UI elements, so first lets load the stylesheet by injecting it into the dom.
            var head= document.getElementsByTagName('head')[0];
            var v_css  = document.createElement('link');
            v_css.rel = 'stylesheet'
            v_css.type = 'text/css';
            v_css.href = 'http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.16/themes/redmond/jquery-ui.css';
            head.appendChild(v_css);

            //Okay, now we need the core jQuery library, lets fetch that and inject it into the dom as well
            var script= document.createElement('script');
            script.type= 'text/javascript';
            script.src= 'https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js';
            head.appendChild(script);
                    
            checkjQueryLoaded = setInterval(function()
            {             
                if(typeof jQuery != "undefined")
                {
                    //Okay, now we need the core jQuery UI library, lets fetch that and inject it into the dom as well
                    var script= document.createElement('script');
                    script.type= 'text/javascript';
                    script.src= 'https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.16/jquery-ui.js';
                    head.appendChild(script);
                    window.clearInterval(checkjQueryLoaded);
                }
                else if(maxLoadAttempts < jQueryLoadAttempts)
                {
                    window.clearInterval(checkjQueryLoaded);
                }
                jQueryLoadAttempts++;
            },300);
            
            jQueryLoadAttempts = 0;        
        
            checkLoaded = setInterval(function()
            {             
                if(typeof jQuery != "undefined" && typeof jQuery.ui != "undefined")
                {
                    window.clearInterval(checkLoaded);
                    callback(true);
                }
                else if(maxLoadAttempts < jQueryLoadAttempts)
                {
                    window.clearInterval(checkLoaded);
                    callback(false);
                }
                jQueryLoadAttempts++;
            },500);
        }
    }
    catch(exception)
    {
        callback(false);
    }
}

Then you can invoke it and have a callback like this

loadJQuery(function(loadSuccess){
    if(loadSuccess)
    {
	//Do your jQuery stuff here. Basically you can think of this as a replacement for your 
	//document.onReady code
        $(document.getElementsByTagName('body')[0]).append("<div id=infoNotice title='Success'>jQuery and jQuery UI loaded!</div>"); 
        $( "#infoNotice" ).dialog({ modal: true});  
    }
    else
    {
        alert('Couldn\'t load jQuery :(');    
    }
});

Have fun!


Salesforce, google maps, and jQuery fun

Another Cloudspokes challenge entry submission. This one was for the challenge http://www.cloudspokes.com/challenges/1345. Basically the idea is get Sales data from Salesforce, plot it on a google map using geocoding without creating duplicate map points and allow a user to click one of the points to see all the sales data for it. The tricky part here is that the data comes from different objects based on what country you are filtering on (oh yeah, it has to support multiple countries). So data for the united states comes from a custom object called Sales Orders, while data for Japan and Germany come from opportunities. It had to allow to be easily expanded for more countries in the future as well with an easy way to set the data source. It was recommended to use address tools (an application from the app exchange that has prepopulated lists of countries and their states, along with some other data) for country and state data, but then there was some confusion because address tools is a for pay app with a 14 day trial. What is a developer to do?

I decided to try and make the best of both worlds. Using a flag in the javascript you can tell the controlling class whether to try and pull country and state data from address tools, or return a hard coded set of data (which was said to be an acceptable alternative in the comments of the challenge). The bummer here is that the org does at least have to have the address tools objects otherwise the class won’t compile. Nice thing is my installable package does include the objects and fields, so while it doesn’t have all the data that a full functionally addressTools would have it should at least install and not error. If you do have a functional addressTools install then no need to worry at all. The application will just work, because it defaults to attempting to pull it’s data from there.

To solve the issue of pulling data from different objects with different field names, I decided to create a wrapper object. A simple class that contains only the data needed to plot the address on the map. So whether the data be originally coming from Sales Orders or opportunities, they both end up returning a list of salesData objects (which is what I ended up calling my wrapper class). I created two separate methods (though they probably could have been consolidated into one, but it would have been a bit messy) for getting data from either object. The correct method was called by another which gets invoked by the user interface. Something like

User picks country
Javascript uses apex remoting to call getSales(string formData);
deserialize the form data from a url query string into a map of string to string (key to value)
find useOpp key in the deserialized data (this got set by the javascript in the application before the request was sent)
call the buildQueryFilter method and pass the form data. This method evaluates the data passed in the form and creates a SOQL where condition that will filter the records as the user has requested.
If useOpp is true call the getOpportunitySales() method. If not, call the getSalesOrderSales() method.
Both methods return a map of address (string) to salesData objects, using the filter created above.
Return the map of addresses to salesData objects to the javascript to be plotted on the map.

Those few parts where really the trickiest part of this challenge. I feel creating the wrapper object was probably the slickest solution, and even allows for other potential data sources in the future, and easily expand-ability to return more data to the front end if desired. I’ll be honest and stay a little bit of my code is redundant because of a feature I added at the very last moment, so I end up deserializing the form data twice, which I should really only need to do once, but it’s a short string of data so it’s not a big deal. I also not 100% sure the application is safe from SOQL injection. You could probably get the application to error by passing junk data with firebug or something, but I doubt you could make it do anything besides just error. I mean SOQL is select only anyway, and the filters it runs through and the way the query string gets built is pretty solid. So I am pretty sure at worst an attack could just get the application to toss some errors for their instance of it. Nothing that should be able to bring the app down, especially with governor limits in place.

As usual, I can’t release the source code myself until I have lost, or Cloudspokes gives me the okay. They generally host all code on their github anyway so in that case I’ll updated this post with the link to it.

Anyway you can see the video here: http://www.screencast.com/t/cLUc7dqpHEkC
Or play with the Demo App!


Salesforce Call Apex Code From Hompeage Component

So this is something I had to do for a challenge recently, where I wanted to invoke my Apex class/method from a home page component. I had heard some solutions involving a link that called a visualforce page (nice thought, but a little clunky and hard to pass parameters) or loading a visualforce page tied to the controller in an iframe (again, works but not the most elegant). I remembered that there is a connection library that allows javascript to call Apex, that is normally used for custom buttons and back in the day s-controls. I putzed around with the syntax some and realized I was making it way more complicated than it needed to be. Below is my solution that I think is pretty slick.

1) Write an apex method that is set as a webservice so it can be called by the library (example below)

    webservice static string createLinkSimple(string url, string title)
    {
          string returnVar = 'Link added!';
          try
          {
               favLink__c thisLink = new favLink__c();
               thisLink.location__c = url;
               thisLink.name = title;
               thisLink.owner__c = UserInfo.getUserId();
               insert thisLink;
           }
           catch(Exception e)
           {
               returnVar = 'Error adding link: ' + e.getMessage();
           }    
           return returnVar;
    }

2) Create a home page link (setup->customize->home-> custom links)
3) Create a link link. Set the content source as ‘onClick javascript’
4) Enter code similar to the following

{!REQUIRESCRIPT("/soap/ajax/10.0/connection.js")}
{!REQUIRESCRIPT("/soap/ajax/10.0/apex.js")}

var result = sforce.apex.execute("favLinks","createLinkSimple", {url: document.location.href, name: document.title});
alert(result);

(The first param is the class name, the second is the method name, and then pass the arguments in the {} brackets)

5) Save it and add the link to a homepage component and display it in the sidebar. You are done!

The will allow you to invoke Apex classes and store the result in a variable to do whatever you want with. Combine this with my Salesforce – Pushing Custom Buttons to the Limit With jQuery and Apex article and you can build some pretty powerful cool stuff.

Hope this helps someone out there.


While This is Awesome, There Must be a Better Way.

This is half an interesting post about how I solved a fairly complicated problem, and half me looking for a better way to do it. The problem I was solving in theory is fairly simple.

I want to import answers from an external survey software into our Salsforce org.

Pretty simple right? Standard import functionality. Nothing too complex. But now let’s build on this.

1) It should be automated. No manual triggering of the import.
2) It should be imported as it is created. Closer to real time than to ETL.
3) The place the data is coming from has no API. We only have direct ODBC access to it.
4) Since the process must be closer to real time, we should use a ‘push’ style mentality for import. The survey software should push the data into Salesforce as it is created, instead of Salesforce requesting it.
5) The survey software only gives me access to javascript to add functionality. No server side processing of any kind.
6) Solution should be as cloud based as possible. Ideally no code on our servers (we have a cold fusion server we use for a lot of data brokering, but we would like to get rid of it).

Suddenly got harder eh? Right off the bat, point 3 means we are going to need to use some kind of middle-ware (in this case our Coldfusion webserver that can talk to the database). Salesforce has no method to talk directly to a database, as it shouldn’t. Everything is done with API’s these days, nobody directly queries databases anymore. At least they shouldn’t. So my first task was to write an API that Salesforce could access to get it’s data. So I wrote up a simple ColdFusion webservice that queries the survey database, formats the data in a reliable way, and returns a JSON array. While not ideal because it does use our server, at least it’s fairly transparent. It’s just a webservice call that could possibly be replaced in the future if the survey software does release an API in the future.

With the webservice up and running, now I need some Apex code to call out to it, get the data and handle it. So I wrote a nice little Apex method that uses a simple HTTP get with some URL params to invoke my ColdFusion webservice. Now it can get the JSON data. Using the new awesome JSON parser in winter 12 I am able to parse the data into a list of custom objects. Hooray.

So now I need a Salesforce object to store this data right? Actually I am going to want a few. I ended up creating 3 in total. The first object ‘survey’, is just a simple container object with pretty much no fields. The second object, ‘survey entry’ is just an object that will contain all the answer objects for a person in a given survey. It of course has a lookup to the ‘survey’ object, as well as a lookup to a contact, and some other info (when they took the survey, etc). The third object ‘survey answer’ is where the real juicy data is. It has the ID of the question, the text of the question, the persons answer, and a lookup to the survey entry.

So now I modified my Apex class a bit to create the survey answers objects, and relate them to the proper ‘survey entry’ (and create one if one does not exist for this person in this survey yet). Boom, so now all the ‘hard’ work is done. I have a class that can be called that will import the survey data for a person into Salesforce. But wait, how do I actually call this thing? I don’t want this on a scheduler, I want it to get called when new data is available. So I need to make this class itself into a webservice of some kind.

I have a bit of experience with Apex REST so I decided that would be a fitting way to handle this. This class only needs the ID of the survey, and the person for whom it needs to import data. That information is easily included in the URL or in POST fields, so I quickly modified my class to be an Apex REST service. Now it was ready to begin being accessed from the outside world. The question now is, how do I invoke the service itself?

First I used the apigee app console to make sure it was working as required. Apigee handles the oAuth and lets you specify params so testing your Apex REST service couldn’t be easier. Once I had verified that it worked, I needed some method to allow the survey software to invoke it. Problem is of course, if you remember that the survey software only supports JavaScript. JavaScript is still subject to that cross domain security policy BS. Normally you could use the script injection technique to make a callout to a different domain, but I need to set headers and such in the request, as well as making it a post request, so that wasn’t going to fly. On top of that I have would have no idea how to let JavaScript start using oAuth or get a valid session ID. So here is where things get a little murky.

How could I allow a javascript only application invoke my Apex REST service? Looks like I would again have to turn to my ColdFusion middleware box. I wrote another webservice which can invoke the desired Apex method from ColdFusion. You can call Apex REST services using a session ID instead of having to deal with oAuth so I went that route. I already have integration between Salesforce and ColdFusion through use of the awesome CFC library provided at RIA Forge (I actually helped contribute a bit to that). So I just wrote up what basically amounts to be a wrapper. You can invoke it from a simple get request and it will wrap the request with the required headers (authorization, content-length, content-type) and send it off to Salesforce. ColdFusion web services have the awesome feature of being able to be called via a URL, instead of having to use a WSDL or whatever. Come to think of it, they are almost a forerunner for REST, but I digress.

So now I have a URL that when called (with some arguments/params) will invoke my Apex REST service that goes and gets the survey data and imports it. So now I still need to get the survey software to call this URL. Here I use the classic script injection technique to make my cross domain request (because the survey software and my ColdFusion box live on different domains) and it much to my surprise it all worked. If you are curious, the code to do that looks like this.


function loadJSON(url)
{
var headID = document.getElementsByTagName("head")[0];
var newScript = document.createElement('script');
newScript.type = 'text/javascript';
newScript.src = url;
headID.appendChild(newScript);
}
var survey = '12345';
var token = 'mkwcgvixvxskchn';
var contact = '003GASDFADFAFDAS';
newUrl = 'http://XXXXXXXXXXXXX/webservice.cfc?method=importData&survey='+survey+'&token&='+token+'&contact='+contact;
loadJSON(newUrl);

So in the end, this is the process I came up with.

1) User takes online survey from 3rd party site (lets call it survey.com)
2) survey.com invokes javascript which calls the ColdFusion webservice (which includes survey id and person id in the request)
3) ColdFusion receives the request, and ‘wraps’ it with the needed authorization information to make a valid HTTP request.
4) Salesforce custom Apex REST class receives the request (with survey id and person id still included)
5) Salesforce sends request BACK to a different ColdFusion webservice, which requests the actual question/answer data.
6) ColdFusion receives request. Queries survey database and encodes database info as an array of JSON encoded objects.
7) JSON encoded data is returned to the calling Apex, where it is parsed into custom objects and committed to the database.

Seems kind of obtuse eh? Yet I can’t think of any way to make it leaner. I really would like to eliminate the 2nd and 3rd step and just have the survey software invoke the Apex REST directly somehow, or at least make the call totally cloud based. I suppose I could host a visualforce page that does the same thing, and have the JavaScript call that…

So anyway, here you can see an interesting case study of integrating a lot of different crap using some fairly new methods and techniques. I am totally open to suggestions on how to refine this. Right now there are just so many points of failure that it makes me nervous but again it seems to be about the best I can do. Thoughts and feedback welcome in the comments.