WiseJive

Proxied - Web Service for the static page web app

2011-03-31T12:02:00.000-07:00

I mentioned in a previous post how I had been building a couple of static page apps that used only Javascript with JSONP to get and push data. These apps I have been working on mostly involved using Facebook and Twitter, and therefore I didn't run into too many problems with the fact that I had no way to persist data (leaving out HTML5 local storage, which doesn't allow aggregation across users anyway). However I have gotten to the point with the projects where I'd really like to be able to do some things that involve aggegation of data across users, and frankly I have not found a webservice to do this (I went through all this in my last post on this subject).

So having thought this over more I think such a service, which I have named Proxied, would have the following:

Login/Registration/Session as a service capability - Proxied would have the ability to establish a user session per Proxied application, so that other Proxied API calls could be limited to logged in users
True to its name Proxied would proxy calls to any api, ensuring they return JSONP
For many API's some sort of authentication is needed, in those cases Proxied would store your API credentials and do whatever signing is needed for that API.

I think a specific example is probably most helpful.

Say you want to build a new HackerNews like web app, you'll need

Login system
Database to store submitted links
Database to store up-votes, down-votes, karma

To do this with a static page web application and Proxied we'd use the Proxied login system to create user sessions and then a cloud database server (say SimpleDB, or MongoHQ) to store the data. We'd create whatever tables (or no tables for NOSQL) we need to in the cloud DB, and then figure out what the cloud DB API endpoint url(s) would be for updating the tables, we'd set those endpoint URLs in Proxied along with our cloud DB credentials, and signing methodology (OAUTH or whatever). Proxied would then produce a JSONP accessible endpoint (or better yet, with clever CNAME's, JSONP wouldn't be necessary) that would proxy the call to the cloud DB API to update our application's data. In Proxied we could set the cloud DB proxy endpoint(s) to be accessible only by logged in users, therefore restraining willy nilly use of our cloud DB by nefarious folks.

Basically one would use Proxied to setup API endpoints for what, in a traditional webapp, would be the controller (in Rails parlance) endpoints on your own server. For a RESTFUL architecture you'd have Proxied endpoints to GET/PUT/DELETE/POST your data to a cloud service. As you add more data tables (or non-tables for NOSQL), you'd add more endpoints in Proxied to service that data.

At this point you can just write the app as a static web page that uses Javascript to update the application data as necessary. Need to crunch data? Well that should be done offline anyway, using Map-Reduce or the like, just update the cloud DB with your crunching results and make that available through Proxied.

Nothing restrains you from using more than one web page, but all the pages would be static. I'd then host the web pages on Amazon S3 thereby obviating the need for ANY server at all on our part, and ensuring the application is scalable (to the degree the underlying services are scaleable).

What I want to know is whether this is a service that developers would use? If so, how much would you be willing to pay for it?

Phonegap, JQuery Mobile, Twitter and Facebook

2010-11-30T22:09:00.000-08:00

Recently I have been working on a mobile website that I wanted to turn into a iOS app. Not knowing (or wanting to know) Objective C, I decided to use PhoneGap to build a HTML/Javascript/CSS based application. My mobile website uses JQuery Mobile, so I wanted to continue to use that framework if at all possible (it offers a lot of easy mobile styling and touch based events). My application also makes substantial use of the Facebook and Twitter APIs, so I also needed to find a way to authenticate with those services (both use a form of OAuth).

The first thing I tackled was the OAuth integration portions. I naively thought that the two services respective Javascript login procedures would work in the WebKit window that PhoneGap creates, however I found neither operated. That meant that I had to fall back on doing actual OAuth. I found a great blog post describing how to do this for Facebook, however it didn't do everything I needed it to do, so I thought I describe what I did in more detail.

The first thing you need to do is download the ChildBrowser Plugin and drag and drop the files under the ChildBrowser/iPhone to the plugins folder of your XCode project. Then put the ChildBrowser.js file in your www folder and make sure to include the file as a script tag in your html files AFTER the phonegap.js file.

<script src="phonegap.js" type="application/x-javascript" charset="utf-8"></script>
<script src="ChildBrowser.js" type="application/x-javascript" charset="utf-8"></script>

The next step is to fire off the following when you want to login to Facebook (like a click event or the like).

function(){
 var my_client_id = "YOUR_CLIENT_ID", my_redirect_uri = "http://www.facebook.com/connect/login_success.html", my_type =            "user_agent", my_display = "touch";

 var authorize_url = "https://graph.facebook.com/oauth/authorize?";
 authorize_url += "client_id=" + my_client_id;
 authorize_url += "&redirect_uri=" + my_redirect_uri;
 authorize_url += "&display=" + my_display;
 authorize_url += "&scope=read_stream,publish_stream,offline_access,publish_checkins"

 client_browser = ChildBrowser.install();
 client_browser.onLocationChange = function(loc){
     facebookLocChanged(loc);
 };
 if (client_browser != null) {
    window.plugins.childBrowser.showWebPage(authorize_url);
 }
});

This basically does the first step of the OAuth flow and shows the Facebook Authorization page int a ChildBrowser window.

The next thing we do is wait for a change in the location of the ChildBrowser window to the redirect_uri.

function facebookLocChanged(loc){
/* Here we check if the url is the login success */

if (loc.indexOf("http://www.facebook.com/connect/login_success.html") > -1) {
 client_browser.close(); 
 var fbCode = loc.match(/code=(.*)$/)[1]
 $.ajax(
  {
     url:'https://graph.facebook.com/oauth/access_token?client_id=YOUR_CLIENT_ID&client_secret=YOUR_CLIENT_SECRET&code='+fbCode+'&redirect_uri=http://www.facebook.com/connect/login_success.html',
     data: {},
     success: function(data, status){
    localStorage.facebook_token = data.split("=")[1];
    client_browser.close();
    initialize_facebook();
     },
     error: function(error) {
    client_browser.close();
     },
     dataType: 'text',
     type: 'POST'
  }
 )
}
}

This completes the OAuth process closes the ChildBrowser (I close the ChildBrowser first because I like the user experience) and stores the facebook_token. Note here we have taken advantage of the fact that Cross-Domain AJAX calls are allowed when using PhoneGap. We can now make Facebook calls. I wrote a little helper method for appending on our facebook_token to urls when making calls.

function create_facebook_url(base) {
if(device) {
 if(base.indexOf("?") == -1) {
  return base + "?access_token=" + localStorage.facebook_token
 } else {
  return base + "&access_token=" + localStorage.facebook_token  
 }
} else {
 return base;
}
}

// Example of using create_facebook_url
FB.api(create_facebook_url("/me"), function(result){
   current_facebook_user = result;
}

For Twitter we follow a similar procedure, though the Twitter OAuth is markedly more complicated. To handle some of the more complex OAuth operations I opted to use an oauth javascript library, so this also needs to be included in your index.html files.

<script src="/javascripts/sha1.js" type="text/javascript"></script>
<script src="/javascripts/oauth.js" type="text/javascript"></script>

Note the sha1.js file is included with the oauth library.

Once we have the libraries included we can now start the OAuth process.

accessor =
{ consumerKey   : "YOUR_CONSUMER_KEY"
, consumerSecret: "YOUR_CONSUMER_SECRET"
, serviceProvider:
{ signatureMethod     : "HMAC-SHA1"
 , requestTokenURL     : "http://api.twitter.com/oauth/request_token"
 , userAuthorizationURL: "https://api.twitter.com/oauth/authorize"
 , accessTokenURL      : "https://api.twitter.com/oauth/access_token"
 , echoURL             : "http://localhost/oauth-provider/echo"
}
};

var message = {
method: "post", action: accessor.serviceProvider.requestTokenURL
, parameters: [["scope", "http://www.google.com/m8/feeds/"]]
};
var requestBody = OAuth.formEncode(message.parameters);
OAuth.completeRequest(message, accessor);
var authorizationHeader = OAuth.getAuthorizationHeader("", message.parameters);
var requestToken = new XMLHttpRequest();
requestToken.onreadystatechange = function receiveRequestToken() {
if (requestToken.readyState == 4) {
var results = OAuth.decodeForm(requestToken.responseText);
var oauth_token = OAuth.getParameter(results, "oauth_token");
var authorize_url = "http://api.twitter.com/oauth/authorize?oauth_token="+oauth_token;
client_browser = ChildBrowser.install();
client_browser.onLocationChange = function(loc){
 twitterLocChanged(loc, requestToken, accessor);
};
if (client_browser != null) {
 window.plugins.childBrowser.showWebPage(authorize_url);
}
}};
requestToken.open(message.method, message.action, true);
requestToken.setRequestHeader("Authorization", authorizationHeader);
requestToken.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
requestToken.send(requestBody);

Basically we are forming the proper OAuth request to get a request_token from Twitter and once we receive back our token we use it to open a ChildBrowser pointing at the proper Authorization page. Next we are (same as Facebook) look for a location change on our ChildBrowser. This time we look for the location to change to the callback that we set in our Twitter Application definition.

function twitterLocChanged(loc, requestToken, accessor){
/* Here we check if the url is the login success */
if (loc.indexOf("http://www.headingtoo.com/") > -1) {
 client_browser.close();
 var results = OAuth.decodeForm(requestToken.responseText);
 message = {method: "post", action: accessor.serviceProvider.accessTokenURL};
 OAuth.completeRequest(message,
  { consumerKey : accessor.consumerKey
  , consumerSecret: accessor.consumerSecret
  , token : OAuth.getParameter(results, "oauth_token")
  , tokenSecret : OAuth.getParameter(results, "oauth_token_secret")
 });
 var requestAccess = new XMLHttpRequest();
 requestAccess.onreadystatechange = function receiveAccessToken() {
  if (requestAccess.readyState == 4) {
   var params = get_url_vars_from_string(requestAccess.responseText);
   localStorage.twitter_token = params["oauth_token"];
   localStorage.twitter_secret_token = params["oauth_token_secret"];
   localStorage.twitter_user_name = params["screen_name"];
   localStorage.twitter_user_id = params["user_id"];
   intialize_twitter();
  }
 };
 requestAccess.open(message.method, message.action, true);
 requestAccess.setRequestHeader("Authorization", OAuth.getAuthorizationHeader("", message.parameters));
 requestAccess.send(); 
}
}

This code is fired when we notice our ChildBrowser has gone to the defined callback url, we then complete the OAuth process by asking for an access_token, which we put in local storage. We can then use that access token to make request on behalf of our user.

// helper
function get_url_vars_from_string(url) {
    var vars = [], hash;
    var hashes = url.slice(url.indexOf('?') + 1).split('&');
    for(var i = 0; i < hashes.length; i++)
    {
        hash = hashes[i].split('=');
        vars.push(hash[0]);
        vars[hash[0]] = hash[1];
    }
    return vars;
 
}

// helper
function create_twitter_oauth_ajax_data(url, method, params, success, error) {
var ajax_data = {
               url: url,
               data: params,
               dataType: 'json',
               type: method,
   timeout: 60*1000,
               beforeSend: function(req){
                   var message = {
                       method: method,
                       action: url
                   };
                   var message_params = params;
    message.parameters = message_params;
    debug_log(message.parameters);
                   OAuth.completeRequest(message, {
                       consumerKey: accessor.consumerKey,
                       consumerSecret: accessor.consumerSecret,
                       token: localStorage.twitter_token,
                       tokenSecret: localStorage.twitter_secret_token
                   });
    debug_log(message);
    req.setRequestHeader("oauth_consumer_key", accessor.consumerKey);
    req.setRequestHeader("oauth_nonce", message.parameters['oauth_nonce']);
    req.setRequestHeader("oauth_signature_method", 'HMAC-SHA1');
    req.setRequestHeader("oauth_token", localStorage.twitter_token);
    req.setRequestHeader("oauth_timestamp", message.parameters['oauth_timestamp']);
    req.setRequestHeader("oauth_version", '1.0');
    req.setRequestHeader("Authorization", OAuth.getAuthorizationHeader("", message.parameters));
                req.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
   },
               success: success,
               error: error
           }
return ajax_data;
}
var success = function(data, responseText) {
current_twitter_user = data;
};
var error = function(error){
debug_log(error['responseText']);
current_twitter_user = null;
debug_log("problem getting twitter user");
};
var ajax_data = create_twitter_oauth_ajax_data('http://api.twitter.com/1/users/show.json', 'get', {screen_name:localStorage.twitter_screen_name, user_id:localStorage.twitter_user_id}, success, error);
$.ajax(
ajax_data
);

Above I show some helper methods I created and a call to get a user's data (which doesn't necessarily need authentication, but demonstrates how it is done anyway).

The final thing that I learned has to do with the current version of JQuery Mobile. I was having trouble getting the app to switch pages when the pages were all in different files. JQuery Mobile is supposed to do an AJAX request for the separate files and then include them in the "one-page-app" that is creates. This didn't seem to be working, so what I did was include all my pages in the index.html page, and stopped trying to use separate files. To do this you just have to build separate divs with each of the pages.

<div data-role="page">

<div id="home_page" data-role="header">
</div><!-- /header -->

<div class="home_page" data-role="content">
 <div id="main">
  <div id="home">
    <div id="logins">
              <span id="twitter_login" class="button twitter"><img id="twitter_login_button" alt="twitter login" src="/images/twitter_16.png"/>&nbsp;Login</span><span class="logout no_show" id="twitter_logout_container" style="display:none"><img id="twitter_avatar" alt="" height="16" src="" width="16"><img src="/images/twitter_16.png" alt='Twitter' title="Twitter" /><a id="twitter_logged_in_link" href="#" target="blank"></a><a href="#" id="twitter_logout" class="logout">logout</a></span>
     <span id="facebook_login" class="button facebook"><img src="/images/facebook_16.png" />&nbsp;Login</span><span class="logout no_show" id="facebook_logout_container" style="display:none;"><img id="facebook_avatar" alt="" height="16" src="" width="16"><img src="/images/facebook_16.png" alt='Facebook' title="Facebook" /><a id="facebook_logged_in_link" href="" target="blank"></a><a id="facebook_logout" href="#" class="logout">logout</a></span>
          </div>
  
    <div class="logo">
     <img src="/images/logo.png"/>
    </div>
    <input type="text" id="where" value="Where are you headed?"/>
    <div id="where_message_container">
     <textarea id="where_message"></textarea>
     <a href="#" id="where_message_submit">Submit</a>
    </div>
  </div>
 </div>
 <div id="fb-root"></div>
 <script src="http://connect.facebook.net/en_US/all.js"></script>
 <script>
    FB.init({appId: '161124300591022', status: true, cookie: true,
              xfbml: true, scope:'publish_checkins'});
 </script>    
 </div>
  <!-- /content -->

<div data-role="footer">
 <div class="ui-grid-b">
  <div class="ui-block-a">
   <a href="#ui-page-start" data-role="button">
    Heading Too
   </a>
  </div>
  <div class="ui-block-b">
      <a href="#/mobile/friends_headingtoos.html" data-role="button">
       Friends
      </a>
  </div>
  <div class="ui-block-c">
      <a href="#/mobile/nearby_headingtoos.html" data-role="button">
       Everyone
      </a>
  </div>
 </div><!-- /grid-a -->
</div><!-- /page -->
</div>
<div id="/mobile/friends_headingtoos.html" data-role="page">

<div data-role="header">
 <h1>Friends</h1>
</div><!-- /header -->

<div data-role="content">
<div id="friends_headingtoos">
   <ul></ul>
</div>
</div><!-- /content -->
<div data-role="footer">
 <div class="ui-grid-b">
  <div class="ui-block-a">
   <a href="#ui-page-start" data-role="button">
    Heading Too
   </a>
  </div>
  <div class="ui-block-b">
      <a href="#/mobile/friends_headingtoos.html" data-role="button">
       Friends
      </a>
  </div>
  <div class="ui-block-c">
      <a href="#/mobile/nearby_headingtoos.html" data-role="button">
       Everyone
      </a>
  </div>
 </div><!-- /grid-a -->
</div><!-- /header -->
</div>
<div id="/mobile/nearby_headingtoos.html" data-role="page">
<div data-role="header">
<h1>Nearby</h1>
</div><!-- /header -->

<div data-role="content">
<div id="nearby_headingtoos">
  <div class="map"></div>
   <ul></ul>
</div>
</div>
<div data-role="footer">
 <div class="ui-grid-b">
  <div class="ui-block-a">
   <a href="#ui-page-start" data-role="button">
    Heading Too
   </a>
  </div>
  <div class="ui-block-b">
      <a href="#/mobile/friends_headingtoos.html" data-role="button">
       Friends
      </a>
  </div>
  <div class="ui-block-c">
      <a href="#/mobile/nearby_headingtoos.html" data-role="button">
       Everyone
      </a>
  </div>
 
 </div><!-- /grid-a -->
</div><!-- /header -->

Basically just adding in DIV's with the proper id's guarantees they will be loaded when a link with a href="#the_id" is clicked. This works with much more regularity than using external files. Since JQuery Mobile is so young I'd guess that this will be a non-issue at some point.

As always ask questions and leave feedback in the comments.

Static page apps -- where is the database?

2010-11-17T12:49:00.000-08:00

Recently I've been doing a lot of work on applications integrating with social networks and came to the conclusion that with the jsonp apis available from Twitter and Facebook there is no need to actually have a database server, a web application server for dynamic html and all the other accoutrements that most web applications need.

This works rather well in a lot of cases (especially because of JQuery and newer frameworks for organizing single page apps, like sammy.js), I've only found 2 main things that frustrate going this way with all apps.

Some very interesting API's do not have JSONP interfaces, this limits what we can work with and therefore what we can accomplish.
There is no way to store data in a centralized database (HTML5 offers local storage options, but that doesn't allow for aggregation of data).

The first point above can be addressed using Yahoo!Pipes to essentially proxy calls and convert other kinds of output (XML, HTML, etc) to JSONP (see this). Yahoo!Pipes is a nice solution but doesn't have a strict SLA, published rate limits and an easy way to upgrade to a pay per request service.

The second point I haven't found any solution for at all. The problem seems to stem from the fact that private keys cannot be hidden client-side, therefore if one want to use something like Amazon SimpleDB one has to proxy the call themselves to keep their private keys stored securely server-side.

I think there is a spot for some cloud service businesses to fill the voids mentioned above (thinking Amazon style pay-per-use pricing model). I quickly mocked up a node.js app that takes a request with a API URL that is supposed to be proxied, gets the API URL, parses it and returns it as a JSONP response. Node.js is perfect for this kind of work because it is so IO intensive. The database issue is a slightly more difficult problem but is solvable as well.

Soooo.... Anyone interested in working on this? Anyone have some solutions I missed? Lemme know in the comments.

Code tests - What do they test?

2010-06-21T21:07:00.000-07:00

So recently I had a recruiter set me up with an interview in a completely different industry (the financial industry) than my native industry (Internet). My understanding was I was coming as a senior, experienced PhD, with a background in making real systems work and scale. I wasn't particularly interested in a new job in the financial industry, but it sounded intriguing and they sounded like they really wanted me to check it out.

Things went swimmingly till I was asked to do a 2 hour timed code test. I balked, saying that I wasn't sure what they were going to learn from it, catching the HR person quite off guard. Long story short, they decided I wasn't "gung-ho" enough for the job, and we both moved on.

Well this whole thing got me thinking, what are code tests good for? I basically see 3 categories of job candidates, senior, intermediate and junior and the usefulness of code tests vary wildly from senior to junior. I think that HR folks need to be trained to handle each type differently. First off identifying the types:

Senior - Years of experience (3 or more, but not hard and fast), example sites (a portfolio), production code to look at. Blog or tweets related to the field.
Intermediate - Years of experience (less than 3, but not hard and fast), some example sites they worked on as a team but not many (a thin portfolio).
Junior - Years of experience (less than 1, right out of school), very few or no examples.

If the HR person can't identify the person then don't give a code test right away, do a screening interview (< 30 minutes)

Once you have the person in/around a category (nothing is black and white) I think you handle all 3 differently.

Senior - no code tests, respect their experience. Get another senior developer to interview (on the phone) to determine if the candidate is experienced, and most importantly has the right experience for the job. Make sure to ask about things that are key to your product/business, example: If you run a high traffic website, make sure to ask about cache-semantics, and/or eventual consistency.
Intermediate - maybe give a code test (I wouldn't). Understand that a code test will be no more useful than a good interview at figuring out the candidate's skill set.
Junior - by all means give a code test. The candidate has no reason to expect not to get a test, they have no proof of their ability.

So I think code tests are useful in some instances, but no matter what I think a good interview (not even a long one) will be able to determine a lot more than a code test. To be honest an interview is easier on the candidate, and the engineer who would have to grade the code test (boring, tedious work). The plus of an interview is that you get to feel out the candidate on culture as well, find out if they are funny, moody, easily frustrated...

One other thing is to consider the ordering of the process. I think a code test (as you can probably guess from above), should not be an initial step. With this finance company I had a 15 minute interview with an HR person and then was supposed to take a test. I had yet to know what the job would be (what tech is used, what Artificial Intelligence is used, if any, what the big problems are, in short, how fun the job would be for me), and I was being asked to devote 2 hours out of my (busy) life to do a timed test (which stressed me out).

So I think the best way to handle the process is to do some sort of screening interview with an engineer first. This is for the hiring company and the candidate! Both will have a much better idea of each other's interest level after that. Remember the candidate is interviewing you as much as you are interviewing the candidate. This especially goes for the highly experienced, highly skilled candidates.

So that is it, I guess that could be classified as a rant, but it is not a very aggressive one...

Twitter @Anywhere Javascript API exposed

2010-06-10T09:45:00.000-07:00

So I have been working on some serious Twitter and Facebook integration on the client-side for a recent project. To do some of the things I wanted to do I needed to use the Twitter @Anywhere Javascript API. The only problem with using the Twitter @Anywhere API is that it is very thinly documented and not likely to be updated soon.

Being the enterprising fellow I am, I pulled down there main API Javascript code from Firebug and put it into Eclipse (with Aptana) highlighted the one line of code (there were no line breaks), did an Apple-Shift-F on the code (fix the syntax), and boom! I now had a nicely formatted Javascript file with the Twitter @Anywhere API. I won't publish it here as that is Twitter's property and I don't want to get into any trouble.

What I will talk about is some of the things I learned about how to use the @Anywhere API; some things that your really need to make it feasible to use at all, even just to test.

First major find was how to pass parameters through the @Anywhere API to the regular Twitter API. The current @Anywhere cheatsheet describes doing calls like:


User.current.homeTimeline()

That by default will only give you back the first 20 items in the current user's feed, it doesn't allow using a "since_id" or a "max_id" parameter to restrict what items are returned, and it doesn't let you do pagination of the returned results.

From looking over the code I noticed you could input options to the call by doing the following:


User.current.homeTimeline({count:200, rpp:3, since_id:171981, max_id:181981});

This would fire off a request to the regular Twitter API that looks like:

http://api.twitter.com/1/statuses/home_timeline.xml?count=200&rpp=3&since_id=171981&max_id=181981

Which is exactly what we want, now we can control what we get back with the same precision as the regular Twitter API.

The next thing I figured out was how to avoid using the type of statements that the Twitter @Anywhere API cheatsheet examples use. For example the cheatsheet uses stuff like this:


User.current.homeTimeLine().first(20).each(
    function(status) {
         alert(status.text);
    }
);

This is great when you want to do something atomically with each status you get back, but absolutely sucks when you want to know when you have completed going through all the statuses. The above example immediately returns, it is not synchronous, which makes it nearly impossible to know when it is done (maybe some closure could do it, but I couldn't figure it out).

To combat this I dug into the Twitter @Anywhere API further and found that I can also pass in options to put my own callback for success and error:


User.current.homeTimeline({count:200, rpp:3, since_id:171981, max_id:181981, success:my_success_method, error:my_error_method});

Now your callbacks will be called and you will see the "data" element passed in to your callback has an "array" element, which contains all the returned results. Now you can go through your results at your leisure, and know when you are done to fire off other methods.

That is pretty much all I have found out thus far, but it is enough to do quite a bit of work with the Twitter @Anywhere Javascript API. As always cheers and jeers in the comments...

Facebook OAuth on Rails..

2010-05-07T12:27:00.001-07:00

So last night I took on trying to integrate Facebook's new OAuth authorization in Ruby On Rails, and in this post I am going to try and give everyone some idea of how to do this.

In the past I have use facebooker to integrate with facebook, so I already had a facebook.yml setup with all the configuration information for my facebook app:


development:
application_id: XXXXXXXXXXXX
api_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
secret_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
canvas_page_name: blah
callback_url: http://localhost:3000/
pretty_errors: true
set_asset_host_to_callback_url: false
tunnel:
public_host_username:
public_host:
public_port: 4007
local_port: 3000
server_alive_interval: 0

The only real difference between this and the normal facebooker file is the addition of the application_id, which is now used in the OAuth authentication.

The first step is to add an endpoint to start the first leg of the OAuth.


def new_facebook
facebook_settings = YAML::load(File.open("#{RAILS_ROOT}/config/facebooker.yml"))
redirect_to "https://graph.facebook.com/oauth/authorize?client_id=#{facebook_settings[RAILS_ENV]['application_id']}&redirect_uri=#{APP_URL}/facebook_credentials/facebook_oauth_callback&scope=offline_access,publish_stream"
end

This will redirect you to Facebook, it sets the "client_id" to the "application_id" mentioned above. This particular redirect also asks Facebook to ask your user to grant some special permissions, namely "offline_access", and "publish_stream" (allow me to access my users accounts anytime, and let me publish to their streams), there are many other permission detailed here. Also note that we provided a "redirect_url" to Facebook which Facebook will redirect to once the user grants us access to their Facebook account. So now we need to create an endpoint for the "redirect_url"


def facebook_oauth_callback
 if not params[:code].nil?
     facebook_settings = YAML::load(File.open("#{RAILS_ROOT}/config/facebooker.yml"))
     callback = "#{APP_URL}/facebook_credentials/facebook_oauth_callback"
     url = URI.parse("https://graph.facebook.com/oauth/access_token?client_id=#{facebook_settings[RAILS_ENV]['application_id']}&redirect_uri=#{callback}&client_secret=#{facebook_settings[RAILS_ENV]['secret_key']}&code=#{CGI::escape(params[:code])}")
     http = Net::HTTP.new(url.host, url.port)
     http.use_ssl = (url.scheme == 'https')
     tmp_url = url.path+"?"+url.query
     request = Net::HTTP::Get.new(tmp_url)
     response = http.request(request)     
     data = response.body
     access_token = data.split("=")[1]
     url = URI.parse("https://graph.facebook.com/me?access_token=#{CGI::escape(access_token)}")
     http = Net::HTTP.new(url.host, url.port)
     http.use_ssl = (url.scheme == 'https')
     tmp_url = url.path+"?"+url.query
     request = Net::HTTP::Get.new(tmp_url)
     response = http.request(request)        
     user_data = response.body
     user_data_obj = JSON.parse(user_data)
     flash[:notice] = 'Facebook successfully connected.'
     @social_credential = SocialCredential.create_or_find_new_facebook_cred(access_token, session['rsecret'])
 end
end

Here we are doing a lot of things. First we make sure that Facebook returns us the "code" parameter, we need this to complete the authentication. Once we have the code, we then need to request an "access_token" from Facebook. This was tricky for me, because the url is an SSL/HTTPS url and normal httparty or net/http tricks didn't work. Eventually I got it worked out as can be seen in the code above. Once we get the "access_token" we are pretty much done, in this case I go ahead and get the current user's Facebook data by hitting "https://graph.facebook.com/me?access_token=XXXXXXXXXXXXX" and then populate an ActiveRecord object with the data I receive (including the "access_token") for later use.

Keeping the "access_token" is important, because in my case it lets me update the users stream whenever I need to (hence the "offline_access" in the initial request).

I know from my Twitter stream that there is work on wrapping the Facebook OpenGraph API in ruby but I don't know where it stands. It is a pretty simple api, and won't require much, but it would be nice to have it abstracted some.

Okay, cheers and jeers in the comments please.

My (new) Apple Manifesto

2010-04-11T19:41:00.000-07:00

So the iPad came out and it is a pretty cool looking, interesting device. Following right on the heels of that was the iphone 4.0 OS announcement with all kinds of goodies and one glaring baddie. The baddie was Apple becoming even more closed by cutting anything that is not developed in Objective C out of the App Store. This change coupled with the random App Store rejections prompted me to rethink my philosophy on buying and using Apple products.

So I have an iPhone, a Mac Book Pro (MBP) and I also have a hackintoshed desktop running Snow Leopard. I like all of them (although for computers, I really just like OS X). I plan on keeping them all. I am also a developer and my philosophy on development is to keep things as open as possible. Therefore the Apple policies really bug me.

My decision (I know you have been holding your proverbial breathe) is NOT to boycott Apple products but to devote no time to building software that runs on their proprietary closed platform. If I want an iPad, or the new iPhone I'll get one, no need for self-denial. I just won't build anything for their systems, everything I build will be a web application of some sort using HTML 5 or whatever is available.

Maybe it is a cop-out, as I might still be feeding the beast, but I am not just going to ignore the best products on the market and by building more rich web applications instead of apps, I'll be opening up the Apple products despite what Apple is trying to do. I certainly will look at all the other options much more thoroughly, as I'd rather be on a more open platform. Lastly, I will definitely hackintosh more machines (I don't expect to buy ANY more Apple laptops or desktops, as I can hackintosh much cheaper and more powerful machines with no loss of OS X goodness).

So that is it, my new Apple manifesto, leave the cheers and jeers in the comments.

JQuery, OS X like finder and Sortables fun...

2010-04-10T17:14:00.000-07:00

So I have been working on a new project where I need to do some advanced web/javascript stuff with jQuery. Mainly what I am trying to do is emulate the Mac OS X finder behavior for an arbitrary tree structure (given that directory and file structures really are just trees). I found something that would help me do it, someone had already build a jQuery tree plugin, but it had its problems.

First off the plugin makes use of jQuery's clone() function to move elements out of an html tree (html tree consisting of ol/ul's and li's) and into separate div's for the various finder windows. I don't mind the using of clone(), but it does screw up any processing that wants to manipulate the elements that were cloned by id. Basically it clones the nodes, so you'll have 2 nodes in the DOM with the same id, and you won't be able to find both with jQuery (it will return the first it encounters). To get around this you can use classes instead of id's but really in the end there should only be one copy of the elements in the tree, and cloning to me seemed like a hack.

To fix it I first started storing as part of finder viewer div the original parent id (old_parent_id) of where the given tree fragment started:


Finder.prototype.selectItem = function(url,noCache,targets){
 var self = this,
  settings = self.settings,
  target = (targets) ? targets[0] : null,
  listItem = (targets) ? targets[1] : null,
  type = (listItem) ? listItem[0].className.match(/(file|folder)/)[0] : 'folder',
  // data will be whatever is found under the current element (listItem)
  data = (url == 'root')
     ? (settings.url) ? null : this.element
     : $('> ul, > ol, > div',listItem).eq(0),//.clone(true),
  url = (url == 'root' && typeof settings.url === 'string') ? settings.url : url;

   if(listItem != null) {
  var oldId = listItem.attr("id");
   } else {
    if(oldId == null || oldId == 'undefined') {
     oldId = "";
    }
   }
   ......
     // Append the new data
 self.appendNewColumn(url,data,[target,listItem],type,oldId);
   ......
   Finder.prototype.appendNewColumn = function(url,data,targets,type,oldId){
   .......
   // Specify new column, and add necessary attributes
 newColumn = $('')
 // Avoid showing the column when it's not yet ready
 // Also, setting display to none makes DOM manipulation a bit faster
  .css('display','none')
  .attr('data-finder-list-id',columnId)
  .attr('data-finder-list-source',url)
  .attr('data-finder-list-level',columnLevel)
  .attr('id', 'data_finder_level_'+columnLevel)
  .attr('old_parent_id', oldId)
  .css('z-index',0); // Keep beneath other columns

Then I made sure that in contrast to before these changes when we were just removing the cloned items, we now are careful about putting the cloned items back where they started.


Finder.prototype.select = function(target,noCache,actionType) {
......
// Remove visible lists which should not be visible anymore
 wrapperLists.each(function(){
  var finderListWrapper = $(this),
   finderListLevel = finderListWrapper.attr('data-finder-list-level');


  if( finderListLevel >= targetLevel ) {   
   $('.ui-finder-list-item.ui-finder-list-item-active',finderListWrapper)
    .removeClass('ui-finder-list-item-active ' + classesActive );  }
 
 
  if( finderListLevel > targetLevel ) {
   debug(finderListWrapper.attr("old_parent_id"));
   if(finderListWrapper.attr("old_parent_id") != null && finderListWrapper.attr("old_parent_id") != 'undefined' && finderListWrapper.attr("old_parent_id").length > 0) {
    debug(finderListWrapper.children().find("ul"));
    var old_parent = $("#"+finderListWrapper.attr("old_parent_id"));
    debug("old_parent: "+old_parent);
    finderListWrapper.find("div.ui-finder-content").children().each(function() {
     try {
          var child = $(this);
          // prepend is important b/c the finder code looks for the first ul/li/div when rendering
          // and therefore this stuff needs to go back to the top of the old_parent's children
          old_parent.prepend(child);
     } catch(e) {
      debug(e);
     }
    });
   }
   finderListWrapper.remove();
  }
 });

and presto we are no longer cloning the nodes and we have the same behaviour as when we did clone(). Note that this code is not thoroughly tested, but once it is I'll submit it back to the jQuery tree guy to see if he wants to incorporate the changes. Also note, that I tried to indicate where everything goes by using function names and "...." to represent, skips, I couldn't really give line numbers as they have all changed in my version.

So now that I had the cloning stuff straightened out I needed to implement drag and drop on the finder so I could drag things from one part of the finder tree representation to another. To do that I setup my ul's to be sortables in the finder code after a new column was added (in order to ensure that all columns are always sortable):


Finder.prototype.appendNewColumn = function(url,data,targets,type,oldId){
.....
//Append new column
// Plain DOM scripting used as opposed to jQuery as it's faster
self.wrapper[0].appendChild(newColumn[0]);

newColumn[0].appendChild($(data)[0]);
$(newColumn[0]).find(".child_container").sortable({
 connectWith: '.child_container',
 appendTo: $(".ui-finder-container"),
 opacity: .5,
    zIndex: 100,
    helper: 'clone',
    revert: true,
    scrollElements: '.ui-finder-content',
 receive: function(event, ui) {
  debug(ui.sender);
  debug($(this));
  debug(ui.item);
  var sender_id = ($(this)).attr("id").split("_")[2];
  var other_id = $(ui.item).attr("id").split("_")[1];
  debug(sender_id+" "+other_id);
  if(sender_id != other_id) {
   if(ui.sender.attr("id").indexOf("child_container_") > -1) {
    $(ui.sender).find(ui.item.attr("id")).remove();
   } else if(ui.sender.attr("id").indexOf("li") > -1){
    ui.item.remove();
   }
  } else {
   debug("attempting to cancel");
   $(ui.sender).sortable('cancel');
   display_message("You dragged a parent onto a child, which would remove the parent and child from the interaction entirely");
  }
 }
});
$(newColumn[0]).disableSelection()

This snippet sets up the child container to be sortable, and sets up some of the options for sortables. I need to work on making this an option in the finder code, as this is quite specific to what I am doing... Key options are the helper:'clone', which must be this way to drag across the finder viewer div's (if helper is not clone the element can only be dragged within its parent div). Also appendTo: $(".ui-finder-container"), is important, because it tells jQuery to append the cloned helper to the upper container that contains all the finder viewer div's, again this is needed so we can drag across the finder viewer areas. Finally connectWith: '.child_container' tells jQuery that we can drop the element on any other sortable list with a class of 'child_container', which all my ul's are classed as in my finder viewer areas.

The last piece to the puzzle is the receive function which calls back anytime a sortable element is dropped on another sortable element. In here I do some work to figure out if the place where the element that is dropped is a child of the original element, and I then revert the change if so. I do this because I don't want to allow an element to be dropped onto itself basically.

The next thing I had to do to get this going was hack up the jQuery sortables code to use more than one element as a drop zone and scroll all the dropzones. I changed the _mouseDrag method to use the connectWith selector to find all the dropzones and loop through them to scroll any that are appropriate:


_determineScroll:function(event, scrollParent, o) {
 if(scrollParent[0] != document && scrollParent[0].tagName != 'HTML') {
  
  if((this.overflowOffset.top + scrollParent[0].offsetHeight) - event.pageY < o.scrollSensitivity)
   scrollParent[0].scrollTop = scrolled = scrollParent[0].scrollTop + o.scrollSpeed;
  else if(event.pageY - this.overflowOffset.top < o.scrollSensitivity)
   scrollParent[0].scrollTop = scrolled = scrollParent[0].scrollTop - o.scrollSpeed;

  if((this.overflowOffset.left + scrollParent[0].offsetWidth) - event.pageX < o.scrollSensitivity)
   scrollParent[0].scrollLeft = scrolled = scrollParent[0].scrollLeft + o.scrollSpeed;
  else if(event.pageX - this.overflowOffset.left < o.scrollSensitivity)
   scrollParent[0].scrollLeft = scrolled = scrollParent[0].scrollLeft - o.scrollSpeed;

 } else {

  if(event.pageY - $(document).scrollTop() < o.scrollSensitivity)
   scrolled = $(document).scrollTop($(document).scrollTop() - o.scrollSpeed);
  else if($(window).height() - (event.pageY - $(document).scrollTop()) < o.scrollSensitivity)
   scrolled = $(document).scrollTop($(document).scrollTop() + o.scrollSpeed);

  if(event.pageX - $(document).scrollLeft() < o.scrollSensitivity)
   scrolled = $(document).scrollLeft($(document).scrollLeft() - o.scrollSpeed);
  else if($(window).width() - (event.pageX - $(document).scrollLeft()) < o.scrollSensitivity)
   scrolled = $(document).scrollLeft($(document).scrollLeft() + o.scrollSpeed);

 }

},

_mouseDrag: function(event) {

 //Compute the helpers position
 this.position = this._generatePosition(event);
 this.positionAbs = this._convertPositionTo("absolute");

 if (!this.lastPositionAbs) {
  this.lastPositionAbs = this.positionAbs;
 }

 //Do scrolling
 if(this.options.scroll) {
  var o = this.options, scrolled = false;
  var currentElements = [];
  if(this.options.scrollElements != null) {
   currentElements = $(this.options.connectWith);
  }
  for(var j = 0; j < currentElements.length; j++) {
   var scrollParent = $(currentElements[j]).scrollParent();
   this._determineScroll(event, scrollParent, o);
  }
  // and do the the helper's scroll parent (default)
  this._determineScroll(event, this.scrollParent, o);
  if(scrolled !== false && $.ui.ddmanager && !o.dropBehaviour)
   $.ui.ddmanager.prepareOffsets(this, event);
 }

 //Regenerate the absolute position used for position checks
 this.positionAbs = this._convertPositionTo("absolute");

 //Set the helper position
 if(!this.options.axis || this.options.axis != "y") this.helper[0].style.left = this.position.left+'px';
 if(!this.options.axis || this.options.axis != "x") this.helper[0].style.top = this.position.top+'px';

 //Rearrange
 for (var i = this.items.length - 1; i >= 0; i--) {

  //Cache variables and intersection, continue if no intersection
  var item = this.items[i], itemElement = item.item[0], intersection = this._intersectsWithPointer(item);
  if (!intersection) continue;

  if(itemElement != this.currentItem[0] //cannot intersect with itself
   && this.placeholder[intersection == 1 ? "next" : "prev"]()[0] != itemElement //no useless actions that have been done before
   && !$.ui.contains(this.placeholder[0], itemElement) //no action if the item moved is the parent of the item checked
   && (this.options.type == 'semi-dynamic' ? !$.ui.contains(this.element[0], itemElement) : true)
  ) {

   this.direction = intersection == 1 ? "down" : "up";

   if (this.options.tolerance == "pointer" || this._intersectsWithSides(item)) {
    this._rearrange(event, item);
   } else {
    break;
   }

   this._trigger("change", event, this._uiHash());
   break;
  }
 }

 //Post events to containers
 this._contactContainers(event);

 //Interconnect with droppables
 if($.ui.ddmanager) $.ui.ddmanager.drag(this, event);

 //Call callbacks
 this._trigger('sort', event, this._uiHash());

 this.lastPositionAbs = this.positionAbs;
 return false;

},

I am going to try and submit this to jQuery as a enhancement to the current scrollables functionality, as I can see this being useful for a lot of folks (in terms of scrolling multiple drop zones).

So that is about it, I now have a working, non-cloning finder tree viewer along with drag and drop working between the finder viewer areas, and even some rudimentary validation (don't drop a element on its own sub-tree). I should probably spend the time to package this up and give it back to jQuery but I just don't have that much time. If anyone is interested in doing this, I'll be glad to help.

Ping me with questions/complaints in the comments...

Snow Leopard Hackintosh How to

2009-10-10T13:46:00.000-07:00

So those of you who follow me on Twitter know I have been working on a hackintosh for a few days now, with limited success... until now!

I have a fully working hackintosh now, and really it wasn't that hard, it is just assimilating information from too many places. I am going to remedy this now.

First things first my hardware:

Motherboard: Gigabyte, EP-UD4P
VideoCard: Galaxy GTS-250 (512MB)

To start follow the directions at lifehacker. I mean follow them to the "T". First time I tried, I used a USB harddrive instead of USB Memory stick, I recommend the stick (much faster to install). If your BIOS looks different, that is okay, key on the following:

Change settings away from IDE to AHCI
Change the boot order to boot your thumb drive first
Change your power settings as in the lifehacker article

If you follow the instructions above, then you should be able to boot into (a somewhat crippled) hackintosh.

Now, your network won't work (onboard at least, a PCI NIC may work), your display will not be very good (no dual monitors and no way to adjust resolutions). You may even get some kernel panics (black code shows up on the side of your screen and the system is completely crashed and needs a reboot).

First things first, lets get the video card working.

The thread I found the good kext (mac equivalent of a driver) for the GTS-250 is here and the actual driver dmg is here.
Also get kext_utility here.
Copy the driver and kext_utility over to your new hackintosh (I used a USB stick to do this).
On your hackintosh, install the kext_utility (open the dmg, drag to Applications)
Open the driver dmg, get the appropriate driver (512MB for me).
Drag the driver onto kext_utility.
Enter your password
Let it install

Now we are going to make sure that when we boot up we will refresh the drivers (kexts).

Under either your USB Hackintosh drive or your Local drive (depending on whether you still need the USB stick to boot or not, see the lifehacker article for details) under /Extra.
Edit the apple.com.boot.plist file.
Under kernel flags (it will be empty), add "-v -f -x32" (not including quotes). For more on boot flags see this blog post. Briefly the flags say , boot verbose (you'll see lots of text), force refresh of driver cache, and run in 32 bit mode.
Reboot your system

Now you should get better video settings (dual monitors, more resolution options). Lets see about getting the onboard network card working now.

Get the kext here.
Copy over to your hackintosh (again using a USB stick or the like).
Drag onto kext_utility, enter password and install.
Reboot your system

After reboot check your network preferences, should be working.

Now sometime during doing this stuff you may get kernel panics (see above). I got them and they were all related to mdwarm or mdworker, which Googling quickly told me was a problem with spotlight (which indexes the disks and obviously was having a problem warming its indices). I don't care about spotlight, and I found out how to turn it off here (note this link is a good resoure, though he installed differently than the lifehacker article). So specifically to turn off spotlight if you are getting kernel panics:

Boot to single user mode (this took me a stupidly long time to figure out, but in the chameleon boot loader, click any key on the boot screen, then highlight your hackintosh, and press the down-arrow key, it'll give you options, one of which is single user, select that one and hit enter).
You'll get a unix #root prompt, mount the hard disk (right above the prompt OSX kindly gives that command verbatim)
Now we can turn off spotlight indexing, type: mdutil –a –I off
Reboot, indexing is now off and I stopped getting panics

So now you should be setup (I was) so start installing all your favorite mac software.

Once you are setup, you probaby want to remove some of your boot flags (see above) from the com.apple.boot.plist file. I left -v in, b/c I like to see what is going on during boot, but you can remove all, except -x32, I suggest you keep that (the kernel is smart it runs any 64bit program in 64 bit mode even with -x32 set).

One last note: the one other good boot flag to know is -x (not -x32 that is different), which boots into "safe mode", if you get kernel panics and what I said above doesn't work, try getting into safe-mode and then fixing permissions (disk utility>verify permissions).

There you have it, I am now watching ESPN360 football with 64 bit cocoa eclipse running while itunes churns away making me a genius list! I'll report back with anymore instability here.

Update:

So tonight I fought with Samba, and I won! Problem actually wasn't Samba at all, but the fact that one more hackintosh patch was needed. When I would try to get Samba to run (or some other programs too), I would get the following (in the logs or command line):

_CFGetHostUUIDString: unable to determine UUID for host. Error: 35

I found the fix here basically grab the file at the top of the page, then drop it in the directory he says. Then, and this is important, open disk utility, and fix permissions (if you don't when you come back up your network will not work). Reboot and boom samba ran just fine.

Heroku and jobs on a timer...

2009-08-01T10:10:00.001-07:00

So in the intervening time since my last post, I started a new project in Ruby on Rails and I have to say I am enjoying the framework . Getting used to Ruby wasn't as hard as I thought although I am sure I am writing Ruby too much like Python :)

One of the great things about using Ruby is the plethora of hosting options as compared to Python Turbogears (where my own server was pretty much the only option). For the current project I am working I chose to take a look at hosting with Heroku which is what I guess I'd call a hybrid PAAS (Platform As A Service). Hybrid b/c it lets you use a database unlike Google App Engine, but it does not give you root access to a machine. Basically Heroku expects you to be using rails, and lets you do all the "railsy" things you would expect like running rake tasks.

Heroku has been great, though there are two things I'll point out for those intersted

Heroku only supports Postgres, so start your project with it! I didn't, but I also wasn't too far along so was able to port my data over from Mysql fairly easily.
If you are running Windows, be prepared for a bit of fight.

(2) may be more of a Rails thing than Heroku, but a LOT of the Rails plugins expect you to be running *nix Point in fact a lot of them expect a full *nix build environment. So, my advice, make sure you are running Cygwin, and specifically make sure you pull down basically all the DB libraries (to build both the Postgres and the Mysql adapters I needed them, and to build Taps, a DB migration library Heroku uses, you'll need Sqllite3).

Nugget: Once you have all these libs, you may get the following error from Cygwin: Unable to remap [libary name], do the following

1.) Shutdown all bash shells and X windows
2.) In command prompt run "ash rebaseall"
3.) Wait to finish
4.) Profit (I mean you should be good to go)

So once I finally got all that Windows specific stuff put to bed I got my application up and running on Heroku, but I needed something that would schedule and run a task every minute. Heroku offers a Cron Job API, but the most often it will run is every hour, which is not sufficient for my needs. However Heroku does offer the DelayedJob plugin for Rails, could I somehow use that to accomplish the same task?

Short answer is yes. DelayedJob offers an input parameter called "run_at" which allows you to not only delay the job but schedule when it should run. By some clever utilization of this we are able to run a job every minute. How? Here's the code.

First off we need to add the first delayed job when the application starts up, so in environment.rb add the following right before the end of the "Rails::Initializer.run do |config|" block.


config.after_initialize do
Delayed::Job.all.each do |old_job|
 old_job.destroy
end
Delayed::Job.enqueue SocialChecker.new(nil), 3
end

This will delete any old jobs that are hanging around in the database (this is important, you'll see why later) and queues up a new job to be run immediately (I didn't set the run_at time for this job).

Now in the "perform" method of my SocialChecker class (that is how DelayedJob works, you must have a perform method in the class you want to enqueue, and that method is called when the job is run) I have the following.


def perform
log_info("Checking: %s" % Time.now)
check()
new_time = Time.now.advance(:minutes => 1)
Delayed::Job.enqueue SocialChecker.new(nil), 3, new_time
end

This little block performs the "check()" method, which is the meat of the things I actually want to do on a one minute timer. The important part is after the "check()" method, here I create a Time object and make it one minute later than the current time. Then I queue up a new job setting the run_at to the one-minute-later Time object. So what this does is it always adds a new job to the DelayedJob queue and that new job is always one minute in the future, so when the "new" job gets executed, it'll add yet another new job to the queue one minute in the future and on and on.... The overall effect is of a scheduled task running every minute (in my case, but just change "Time.now.advance(:minutes => 1)" to whatever you need.

So lets talk potential problems. First off I think one can see why I added the code to delete leftover jobs on initialize, because if there are leftover jobs, when they run each of them will add a new Job for a minute later so we could get into a situation where we would get a Job explosion. The deletion code just keeps us clean on startup. To be safe one could add that code block into the perform method as well, to also make sure we are clean before we add another job.

One other problem is the potential failure of the perform task (assuming the error is not caught, which of course it should be), one might think this would mean that our future jobs wouldn't get pushed into the queue and we'd lose our scheduled task. That is partially right, if the perform task keeps failing then we would not push a new Job into the queue, but one of the great things about DelayedJob is that it will retry your Jobs with an exponential backdown. So as long as your job doesn't continuously fail (which of course is a much bigger issue), the schedule task should recover, eventually.

Note that to use background tasks in Heroku one must pay $15/month and Heroku will give you "1 thread" to consume your background jobs. So if you have a lot of scheduled jobs in mind then they might conflict this might not work that well. My hope would be that eventually Heroku will allow us to purchase more threads to consume the jobs, allowing this process to scale.

So one thing I am bit worried about is, because of their interesting mechanisms for sharing load among servers, whether Heroku wants people doing these short time-scale Jobs. I am heartened by their documentations saying that one gets a "thread" to perform your Jobs, so my assumption is once they give you a thread it'll hang around and you may as well use it. Hopefully someone from Heroku will get back to me if there is a huge problem with this, it is not my intention to bypass their restrictions, at least not if they are there for a good reason ;)

Hope this helps people looking for some Heroku info, please leave some comments or twitter me @jostheim to help improve my code above.

Twitter please deal with shortened urls...

2009-02-26T12:50:00.000-08:00

So this is a pseudo-rant that really isn't a rant, but this is really annoying me. Twitter doesn't resolve urls when they are presented in the twitter interface. This is really annoying. Why you ask?

Well for one I am never sure if one of my friends is posting something from their own blog, or some other place I am interested in, or they are posting about something completely unrelated/unintresting. I know, I know the message is supposed say something about the url, but it doesn't always, and it certainly doesn't let me eliminate sites I don't like (say I don't like Mashable, not true of course, I can't know that one of my friends just posted a Mashable link).
It is huge security problem. The links are anonymous, so we have no idea where they go. As a web professional I can identify some weird links just from there look, but perhaps more importantly I don't think many of the automatic url scanners in the browsers (for phishing, etc.) actually resolve the url to its source, and therefore don't identify malware sites which have been shortened.

So what should be done... Very simple, for every url that gets posted to twitter they need to do a HTTP Head, and resolve the url to it's source (re-searchr does this for most of the shortened urls it sees). Yes this is another step. Yes it is more load on there servers. Yes it still should be done. Note this will NOT effect the 140 character limit, use as short a url as you want in the message, only when the message is displayed should it be expanded.

The other half of this issue is for the Twitter search, try doing a search for http://www.techcrunch.com on Twitter's search engine. You'll find many fewer results than there really are... that is unless you start getting the shortened versions of the Techcrunch url. Before you say, well okay I'll get the tinyurl and bit.ly url and a few others and be done with it (and with the explosion of shortened url services, good luck with finding the best shortened urls to use)... Twitter search has a 140 character limit, so good luck doing more than 2 shortened urls. Additionally good luck doing a wildcard search for http://www.techcrunch.com/* to get all techcrunch posts urls as well, this is impossible with these anonymized urls that hide the actual url structure.

So there is my rant, and my evidence, let me know what you think either in the comments or @jostheim on twitter, but for goodness sake don't include a shortened url!

Searching Socially (not Social Search)

2009-02-09T09:15:00.000-08:00

As an addendum to my previous post about social search I wanted to talk about one of the other things my project re-searchr.com does which is search socially. First off I had better come clean and explain what the heck the difference is between social search and searching socially. Here is what they mean to me (and re-searchr):

social search: using your social networks explicitly to find the answer to a query. Example, asking questions to your friends.
searching socially: using your social networks to influence standard web searching. Example, re-weighting search results based on how many times your friends socially bookmarked sites in the search results

Social search is very explicit, while searching socially can be much more subtle.

I think that searching socially is going to be a big deal in the future. The best filter for content that one has is through trusted sources, why do I listen to This Week in Technology, because they have proved that I can trust their judgements. Why do people listen to Warren Buffet? Because he has proven that he is highly trustworthy in the field of investments. Trusted sources for the bulk of us (who aren't Robert Scoble) can be found in our social networks. They are people we have made the explicit decision to listen to, probably not because we think they are stupid and untrustworthy, more likely because we value their opinions.

Once we have established who are trustworthy friends are we can start to mine our relationships and apply it to search. The very simplest example is to take a default set of search results form google and find out how many times a given search result url has been mentioned by our social networks. That immediately establishes more value for results that are mentioned more than for results that are mentioned less. Why? Well, not simply because the results were mentioned more, but because they were mentioned more by people we trust.

This simple example is just the beginning we can start to mine sentiment out of our friends comments and then define not just mentions but good and bad mentions for re-ranking. We can also look at how many times a domain was mentioned for more aggregate rankings. We can expand out a shell around our friends, going layer by layer through the social graph for more information.

The important thing to note here is that searching socially will require nothing more from the searcher, than having affiliated social networks and giving access to them. This kind of interaction is ideal as most users (self-included) are lazy and do not want to have to do much work to start getting advantages.

To summarize, social search and searching socially are different, and re-searchr is trying to do both.

Minnedemo

2009-02-07T21:26:00.001-08:00

Some video of me demoing my project re-searchr at Minnedemo.

Social Search and re-searchr

2009-01-17T14:47:00.000-08:00

I am writing this post to clarify my thoughts on the social search space where my project stands among all the others out there, what social search means, and my philosophy on where it should go.

So to start where does social search stand? Well we have some engines, Mahalo, Wikia, etc (help me out add links the comments). We have some platforms, Yahoo SearchBoss, Google Custom Search Engines that well not social search engines are capable of being extended socially. Then we have plays by the major search engines, namely Google Search Wiki. Lets try and review these in turn.

The Sites

Mahalo

Mahalo is a social search engine in that it lets users of the site build "pages" around search results. Users of Mahalo can add links, "fast facts", categories and more. Mahalo recently introduced Mahalo Answers, a system by which users can ask questions and Mahalo users can respond (very similar to Yahoo Answers), there is a payment system in Mahalo dollars that can encourage the answering process.

Mahalo's focus seems to be around the "common query" space. If they can define good pages around common queries then they can handle the bulk of the search traffic and show human parsed, edited and validated links, that in theory should be better than any algorithm for finding good content related to a query. The Mahalo Answers system ties into this by letting users find out about specific things not covered by the common query space or not covered on a Mahalo results page.

Wikia

Wikia has much the same focus as Mahalo, building wiki's around common queries. Wikia does not currently have an answer system built into it. Everything else said above for Mahalo essentially applies.

The Platforms

Yahoo Search Boss

Yahoo search boss is not a social search site, it is an API into Yahoo's search system that allows social features to be built on top of it. OneRiot is a good example of this. OneRiot is a site where you can see the "popularity" of a given search result overlayed onto the result. The whole site is driven by the Yahoo Search Boss API.

To get to the most basic crux, the Yahoo Search Boss API allows 3rd parties to start appending meta-data to the Yahoo results, including social aspects.

Google Custom Search Engine

Google Custom Search Engine's are a different beast than the Yahoo Boss system. Google CSE's allow builders of the CSE to limit the web that is being searched over. While not social to start, this simple concept allows 3rd parties to build CSE's for specific users, essentially socially limiting their search. I had a project about 1 year ago that did this, called iPrecis.com, it turns out to be relatively hard to figure out how to limit a person's search, without a lot of setup by the user (in my opinion).

Big Player's Endeavors

Google Search Wiki

Google Search Wiki is a recently introduced "overlay" on the Google search results page. Basically it allows you to "move" results around for your own searches. If I did a search for "James Ostheimer" and my blog came in at the bottom of the page, I could push it to the top. I could also write a comment about the search result right on the search results page.

For now Google is not using this user defined info for much (like changing search results ordering). In the future they may integrate this information into their algorithms.

My Thoughts

I think that all of these products have their place. I like the crowd sourcing idea behind Mahalo and Wikia. Yahoo Search Boss is creating a lot of innovation with it's full API into Yahoo search. Google Custom Search Engine, well having a lot of potential, I think is turning out to be somewhat of a dud. It simply hasn't grabbed hold of being a great platform to build on, mostly, imho b/c it does not allow the reuse of search results in a full fledged manner as Yahoo Search Boss does. Google Search Wiki seems innocuous to me until Google decides to actually use the user data in a meaningful way.

This all being said, I think there is a huge hole here. All of these services put the macro "crowd" into use here, while ignoring the micro "crowd". There has been a rapid emergence of social networking sites that are part of the "live web". Facebook, Twitter, FriendFeed, Jaiku, Identica... all of these sites are about live or near time interaction with a users social network. The micro "crowd" to an individual user is his/her social networks as defined by whatever services he/she is on. These micro "crowds" can be ten to hundreds to thousands of people, and the key to this "crowd" is that they are trusted more than the macro "crowd" of anyone on the Internet.

An example will help illustrate this: Lets say you want to get a new mortgage for your house, you have questions about what type of loan to get (apropos I know). You ask a question on Mahalo about this and get a number of answers, some answers say get an ARM, some not. You ask your Facebook network the same question, you brother says "if you get an ARM I won't speak to you anymore". Which do you go with?

So yeah, the above is contrived, but another example, you want to know the best way to fix a memory leak in some Javascript code. Would you go with a stranger on Mahalo with no knowledge of you or your skill level? Or do you go with the guy you work with who you know is a genius?

This is the power of micro "crowds", TRUST. You trust your friends, co-workers, family, and even acquaintances more than you trust some random person on the Internet.

This is a hole re-searchr is trying to fill. It will try to tie into as many social networks as it can, and become your glue for getting answers to questions. Notice I said glue, re-searchr is not trying to supplant your social networks. To use a computer analogy, your social networks are the operating system, and re-search just wants to run on top of them.

Right now re-searchr is actually doing more than this, because it was originally conceived as a platform for re-analyzing search results. I am right now trying to figure out whether that blends into the micro "crowd" concept or whether I need to split out these products (comments please!).

DRY Information Architecture

2009-01-13T13:02:00.000-08:00

So today at my job I heard from one of the front-end guys that the new Information Architecture involved repeating a subnavigation several places on a single page. This got me thinking, I, as a developer know that I should do my best to stay DRY (don't repeat yourself), does this same axiom apply to Information Architecture/Design?

I think it does. As a developer when I see a pattern form, I know that it is forming because of some end goal that I am needing to produce many times, most likely this is a useful end goal. I then refactor/rearchitect the code to extract the pattern, make it reusable and extensible (well not always, but if I have my druthers I do).

The same should go for design. If we see that we are doing a sub-navigation 3 times on a single page, then don't we think that maybe our whole page might need some work? Why is the user experience such that the nav has to be 3 places? Is the page too long? Is there too much on the page? Would one sub-nav suffice but it is not big enough or doesn't stand out enough?

My theory (the Ostheimer DRY Design Theory) is that IA/Design are confined to the same DRY theory as software engineering: if you are repeating yourself a lot and that seems to be the only solution, then something bigger is wrong.

Somebody make IE extensions easier!!

2009-01-13T12:06:00.000-08:00

I have written a number of things in the Firefox XUL extension environment, some of which I keep to myself, some of which I publish. When it comes to actually trying to build a product around an extension (which is a hard sell anyway), having just a firefox extension is a non-starter. For my latest project re-searchr.com I am exactly in this position, I have a Firefox extension that I have spent a lot of time on, which works fairly well. However since Firefox is only 22% (and rising) of the browser market share, I can't really get all that far. I need an IE extension.

IE extensions are difficult, you have to have Visual Studio .net and you have to know C# or VisualBasic, Visual C++ or [insert microsoft specific language here]. I don't know any of these languages and do not use Visual Studio, so I am at an extreme disadvantage. It would take a long time for me to climb the learning curve to make a very good IE extension. Ideally Microsoft would see this as a bad thing and come out with a more web developer friendly api to IE, but that is not likely to happen.

So I need work arounds. My first step and one that would have been a good idea anyway is to port my Firefox extension functionality to a bookmarklet. My re-searchr toolbar wants to capture page load events, which is impossible with a bookmarklet, but if I treat the bookmarklet click as a load event I can follow a similar process as the Firefox extension to produce inline analysis to a users search results page (this is what re-searchr does).

This actually went fine, the only hiccup being cross-site scripting (which in the extension sandbox in Firefox is NOT a security problem), which I solved using JSON and adding "SCRIPT" tags to the page instead of using XMLHTTPRequests. So now I have partial functionality in a bookmarklet, this is a good thing. Why? Well a lot of people don't want to download toolbars, so if you can build a bookmarklet that shows off what your toolbar will do (at least partially) then folks can "try before buy" your toolbar, see if they like the results and then download the toolbar.

I digress, I still want a IE based extension for re-searchr, and now I have a set of code in a bookmarklet that will properly run in IE (all Firefox specific extension stuff is removed). This should not be all that hard, what I need is a IE BHO (browser helper object) that will, at the very least, add my bookmarklet javascript onto a page and run it's "start" function on page load. This seems like it should be trivial, and if someone could write this BHO then web developers could develop things as JS/HTML/Flash based web apps and plug them into this framework (which just loads javascript onto a tab's page) and vwhala we have an extension environment for IE. For bonus points:

If the BHO could also set the security constraints to allow cross domain XMLhttprequests in the context of the javascript that BHO loads (not sure if this is possible) then that would be bonus points, and would really allow for a lot more functionality
The BHO allow attachement to page load events, so that the developer could detect if the page loaded is something he wants to play with (to remove ads, or change content), and then either exit, or load the rest of the javascript (so as decrease the load on the browser, load a bit of js to determine whether to load the rest).
The BHO, could become a full fledged javascript API into the Microsoft IE API, this would ideal and awesome.

This seems like an absolute no brainer, download some prebuilt BHO code, with a properties file specifing where the javascript files live (urls, or maybe files in the install package), and the initial callback, fill in the properties, package, and use. I'd pay a $100 one time fee to get the code that I can then distribute as my extension.

So why hasn't this happened? Is this really hard? Maybe someone will read this and give me some feedback.

Django quickstart

2009-01-08T17:47:00.000-08:00

So I have been expirementing with Django as a replacement for Turbogears, and as in my previous post Django has been giving me trouble with quickstarting apps. So I did finally get some stuff rolling and I thought I would share it with the world.

First off, to setup the registration plugin, follow this prescription to the letter. Don't easy_install django-registration, that installs the code in your site-packages, we want to put it in your project so you can customize the views.

Definitely setup a base.html (master template). Create a directory /project_root/templates. Put your master template in /project_root/templates/base.html.

For my base.html I stole the basic turbogears one with some of my own edits:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>{% block title %}My amazing site{% endblock %}</title>


<style type="text/css">
    #pageLogin
    {
        font-size: 10px;
        font-family: verdana;
        text-align: right;
   float:right;
   margin-top:5px;
   margin-right:5px;
    }

    a.logo_image img{
     text-decoration:none;
     border-right:none;
     border-left:none;
     border-top:none;
     border-bottom:none;
   float:left;
   padding:5px;
    }

  ul.menu_list {
   margin: 0px auto;
   list-style-type:none;
   margin-bottom:50px;
  }
  ul.menu_list li {
   display:inline;
   padding-right:115px;
  }
  
  ul.menu_list li a {
   text-decoration:none;
   font-weight:bold;
   font-size:110%;
  }
  ul.footer_list {
   margin: 0px auto;
   list-style-type:none;
   padding-top:5px;
   margin-bottom:20px;
   font-size:140%;
  }
  ul.footer_list li {
   display:inline;
   padding-right:15px;
   padding-top:5px;
  }
  
  ul.footer_list li a {
   text-decoration:none;
   font-weight:bold;
  }
  
  #login_submit {
   text-align:center;
   padding-left:70px;
  }
  
  div#menu {
   margin-left:20px;
  }
  
  li#first {
   padding-left:20px; 
  }
  
  li#last{
   padding-right:20px;
  }
  
  div#loginBox a {
   padding-bottom:10px;
  }
  
  a.login{
   padding-right:5px;
  }
  
</style>

  <link rel="stylesheet" href="/static/css/style.css" />

</head>

<body>

<div id="header">

  <a class="logo_image clear" href="/">
  <img src="" alt="" />
  </a>
  <div id="pageLogin">
      <span>
   <a class="login" href="" >Login</a>
    </span>
  </div>
 </div> 
<div id="main_content">
 <div id="menu">
  <ul class="menu_list">
   <li id="first"><a href="">home</a></li>
   <li>
    <a href="">menu1</a>
   </li>
   <li><a href="">menu2</a></li>
   <li><a href="">menu3</a></li>
   <li id="last"><a href=""> menu4</a></li> 
  </ul>
  </div>

     <div id="content">
         {% block content %}{% endblock %}
     </div>

 <!-- End of main_content -->
</div>
<div id="footer" >
 <ul class="footer_list">
  <li>Copyright &copy; 2008</li>
  <li><a href="">Terms of Service</a></li>
  <li><a href="">Terms of Service</a></li> 
  <li><a href="" >Company</a></li>  
  <li><a href="">Contact</a></li>  
 </ul>
</div>
</body>
</html>

Okay so if we did the registration and setup this base.html, we have the beginnings of a quickstart. Ideally we would then port all the registration templates that turbogears has over to django, but I haven't had time to do all that.

As I port the registration templates over I'll post more here.

Mock Testing = Evil?

2008-12-18T22:06:00.000-08:00

I guess today is get off my chest day.

I don't like mock testing. We do it at my job and I pretty much think it is worthless. Here is my problem with mock testing in a nutshell... what are you actually testing?

As far as I can tell you are testing that you can read request parameters, and then test whether the flow of the controller is correct for given parameter sets. That has some uses, but usually is not where bugs are found (b/c we usually can test these things by viewing the page).

More often than not the problems turn out to be in the middleware somewhere, down in the depths where the data is pushed through some business logic and something goes wrong. Not only are these the bulk of the problems, but they are also the hard problems to find, diagnose and fix.

Lets face it, if the controller can't to the right thing based on the parameters that is going to be pretty obvious. But if you are sorting by the wrong date down in the depths after a bunch of data massaging, well that is going to be hard.

I should mention by the way that determining that the parameters match the right flow is a good thing, but why not push the test out to a higher level, instead of mocking the objects, use real objects and check the produced html for expectations. This test buys you a lot more, it checks the flows working, but also checks your rendering. While a mock test, would only check one of these things and always takes time and maintenance to setup because YOU maintain your mock objects, while in a html unit test your controller (you know the thing you actually are trying to build!!) does the mock stuff for you.

I'll end this rant with this. If the goal of testing is to expose bugs before they get to your QA team, then mock testing is not the best use of resources.

Django and Turbogears: The registration problem

2008-12-17T10:05:00.000-08:00

So I downloaded django a couple weeks ago to get some experience with before I go and get my non-profit on at the http://overnightwebsitechallenge.com (where I'll be working in django). I normally work in turbogears and have the turbogears registration egg installed already for use on my other projects.

So here is problem, I installed the django registration eggs, and wanted to get started with users and registration on django, but I kept getting errors with the import of the registration in settings.py.

Well long story short, the problem turns out to be that both django and turbogears want to use 'registration' as the name of the module to import. With that being the case, whichever egg was installed first will be importing first and therefore be used. In my case that was the turbogears egg, b/c I installed that first (a long time ago). So onto the fix... in manage.py:

import sys

sys.path.insert(0, "yourpathhere/django_registration-0.7-py2.5.egg")

That will ensure that the django registration package is pulled in at the top of your python path and will be used instead of the turbogears one.

Problem solved. After 5 hours.

Starter Post....

2008-12-16T08:38:00.000-08:00

Consider this a warning, this blog is going to be mostly about technical things I run into during my working day (which is pretty much 24 hours a day). There may be an occasional political post, but for the most part expect python, java and other goodness (well if we are talking Java then badnesss) to pervade.