Crawling a website with python scrapy

Think of a website as a web, how do we crawl that web? Chances are you went to that navigation menu and found a link that you found interesting you clicked on that and you went to that page to find that important information that you were looking for. Or probably your favourite search engine did it for you. How did your search engine did that or how can you make that traversal automatically? Exactly thats where crawler comes into business. Chances are your search engine started crawling on your website based on a link you shared somewhere. We will create one such crawler using python’s crawling framework called scrapy. For last couple of months I have been using it, so felt like it would be wrong not to have a blog about it.

It is always better to have a python virtual environment, so lets set it up:

$ virtualenv .env
$ source .env/bin/activate

Now that we have a virtual environment running, we will install scrapy.
$ pip install scrapy

it has some dependency, like lxml which is basically being used for html parsing using selectors, cryptography and ssl related python libraries will also be installed. Pip takes care of everything, but when we will be writing codes, we will see this in our error message very often, so it is always good idea to have some idea about the dependencies.

Now that we have it installed, we have access to few new commands. Using these commands we can create our own scrapy project, which is not strictly necessary but still I personally like to have everything bootstrapped here the way the creator wanted me to have, that way I could possibly have the same code standard the author of scrapy had while writing this framework.

$ scrapy startproject blog_search_engine

It will create bunch of necessary and unnecessary files you can read about all of those files at documentation, but the interesting part here is that it will create a configuration file called scrapy.cfg , which empowers you with few extra commands. Your spider resides inside the other folder. Spiders are basically the BOT that contains the characteristics defination of that BOT. Usually you can create a spider using following command as a solid start:

$ scrapy genspider wordpress wordpress.com

It will generate a spider called wordpress inside your blog_search_engine/blog_search_engine/spiders/ directory. It creates a 4 or 5 lines of code at your file which does nothing. Lets give it some functionality, shall we? But we don’t know yet what we are automating. We are visiting wordpress.com and we will find the a links of an article, and then we will go that link and get that article. So before we write our spider we need to define what we are looking for right? Lets define our model. Model are usually stored inside items.py . A possible Article might have following fields.

class Article(scrapy.Item):
    title = scrapy.Field()
    body = scrapy.Field()
    link = scrapy.Field()

Now we will define our spider.

class WordPressSpider(scrapy.Spider):
    name = 'wordpress'
    start_urls = [ 'www.wordpress.com' ]

    def parse(self, response):
        article_links = response.css("#post-river").xpath("//a/@href").extract()

        for link in article_links:
            if "https://en.blog.wordpress.com/" in link:
                yield scrapy.Request(article_url,
                                     self.extract_article)

    def extract_article(self, response):
        article = Article()
        css = lambda s: response.css(s).extract()
        
        article['title'] = css(".post-title::text").extract()[0]

        body_html=" ".join(css('.entrytext'))
        body_soup = BeautifulSoup(body_html)
        body_text = ''.join(soup.findAll(text=True))


        article['body'] = body_text
        yield article

As we had configured at our scrapy settings yield at the parse hands over your article to pipeline, as it looks, pipeline could be a great place for database operations. This is possibly out of the scope of this particular blog, but yet you can have an outline of what you might need to do if you are using sqlalchemy as database, although sqlalchemy won’t be particularly helpful to deal with what we intend to do here, still i felt it would be helpful to have them.

class BlogSearchEnginePipeline(object):
    def process_item(self, item, spider):
        # a = Article(title=item['title'],body=item['body'])
        # db.session.add(instance)
        # db.session.commit()
        print 'article found:', item['title'], item['body']

        return item

Now we have a spider defined. But how do we run it? Its actually easy, but remember that you need to be inside your scrapy project to make this command work!

$ scrapy crawl wordpress

On the side note scrapy actually provide us options to pass parameters from commandline to pass argument to spider, we just need to define an intializer parameter

class WordPress
        name = "wordpress"
        ...
        def __init__(self, param=None):
                pass
        ...

Now we could call:

$ scrapy crawl wordpress -a param=helloworld

In this blog I tried to give you an outline of database orms. Sofar we have a spider but this spider has no great use so far, we will try to create a search engine with this spider at my next blog. Databases that sqlalchemy deals with are not particularly super good with text searches elastic search could be a great option if we are looking forward to implement a search option so at my next blog, I will be writing about a basic search engine implementation using elastic search. Thats in my todo list for this weekend.

How to add flask-admin to a Blueprint?

For those who works with closed source tools you won’t understand the freedom that we have on opensource tools because in real life our requirements changed over time, the tool that we are using they also start to grow and start to cover things they did not had in mind when they started the project. As a developer, we want our tools to do different things and as a person to person and project to project, we may have different sense of beauty and different meaning of code organization phylosophy. In computer science in general we always try to map our problem with a known solution that we have already solved before. So when you have access to source code of your tool, you can easily dig up that source code extend or alter the functionality and map your solution that matches your situation.

For example while working on python flask framework after couple years I realized how big they have grown over time, they do pretty much everything django is capable to do and it is even better because of their sense of modularity and flexibility. So for this project I am working on I started using flask, flask-admin for administrative panel and I am using flasks blueprint to separate different components of my project. Flask admin is actually not very comfortable or easy to attach with blueprints, that actually makes sense because because it adds admin panel and admin panel should be attached with the main app rather than a sub app like blueprint. But I actually had different use case and with admin panel I had to add my custom views which I don’t want to put at my app.py rather I want it to be in my controller. Other class architecture I had in mind will cause a circular dependency which I always get me in panic. I may not be very much neat and clean, pretty, tidy, person in personal life, I know I have limitations but I try to keep my code pretty and tidy and a thing of beauty that made me dig up the source code of those libraries at my office hours to rewrite this. Enough talk, if Linus Torvalds visits my blog ever he is going to get real mad at me for talking too much. So here you go, my code that I am using that satisfies my need:

# admin_blueprint.py

from flask import Blueprint
from flask_admin.contrib.sqla import ModelView
from flask_admin import Admin

class AdminBlueprint(Blueprint):
    views=None


    def __init__(self,*args, **kargs):
        self.views = []
        return super(AdminBlueprint, self).__init__('admin2', __name__,url_prefix='/admin2',static_folder='static', static_url_path='/static/admin')


    def add_view(self, view):
        self.views.append(view)

    def register(self,app, options, first_registration=False):
        print app
        admin = Admin(app, name='microblog', template_mode='adminlte')

        for v in self.views:
            admin.add_view(v)

        return super(AdminBlueprint, self).register(app, options, first_registration)

#app/admin/controller.py
from admin_blueprint import AdminBlueprint
from common.models import MyModel, db

app = AdminBlueprint('admin2', __name__,url_prefix='/admin2',static_folder='static', static_url_path='/static/admin')
app.add_view(ModelView(MyModel, db.session))
#app/__init__.py

from flask_sqlalchemy import SQLAlchemy

# Define the WSGI application object
app = Flask(__name__,template_folder="../templates",static_folder="../templates")

from app.api.controllers import app as api
from app.frontend.controllers import app as frontend
from app.admin.controllers import app as admin



# Register blueprint(s)
app.register_blueprint(api)
app.register_blueprint(frontend)
app.register_blueprint(admin)

# replacing the following code that I had
#from flask_admin import Admin
#from flask_admin.contrib.sqla import ModelView
#from common.models import *

#admin = Admin(app, name='microblog', template_mode='adminlte')
#admin.add_view(ModelView(MyModel, db.session))


Interesting Swift: overview of some interesting things in swift

So I am playing with swift a little bit for this holiday, and found out it to be very interesting language to play with, they actually designed many things that is actually more interesting than the languages I am very used to with. This blog is possibly not going to be a blog that I would refer people to read for learning as I am mentioning java, python or ruby all the way I am going with this blog and not everyone is supposed to be aquatinted with these language, but I am doing it get away with the feeling that I had while playing around.

So far I could find two ways in swift to initialize variables either var or let. var is for variables that actually varies and let is something that remains the same forever. Java has its constant using final that prevented us from changing the variable values once it got initialised. Ruby ensures it by putting its variable name all capital, which I am not a great fan of. Python does not provide much straight forward support for constants. I personally like constants because I believe internally at programming language implementation level it should give compliers and interpreters enough power to optimize and it helps me to reduce bugs in my code as I would have full control over the code on what I am doing and what the variable is intented to be doing even after passing my code to some one else to modify or add more features. Not to mention I have every freedom whether to define a variable with an associated type or let the initialized value to infer its type, which was also great. Although it is considered as a bad coding style to let swift not to infer the type.

var text: string = "hello world"
var text = "hi world"

lazy vars are also interesting, I actually never thought of something like that as I thought compilers or other optimizations tool we are using would take care of that. When we declare a variable as lazy variable, swift initialize that variable when it is absolutely required.

It is also interesting and very useful I believe the way swift syntax is taking care of if variable is initializaable or not.

if (let text = button.currentTitle){
 //display.text = display.text + text  
}

Python has a different syntax idea about ternary operations where java, ruby has almost the same syntax as swift.

output = a != nil ? a! : b

Some short hands on ternary operations comes handy like ruby:

output = a ?? " "

This is interesting that swift won’t allow me to put a variable without we initializing it. Or we have options for definiting a variable as optional when we are not intending to initialize it, now what are optionals? Glad that you asked.

Optionals are actually very interesting concept to me, In ruby code, while using other peoples API send? and send! with a bang, these variations of name actually told me a lot of details and safetly measure required or not on that methods by convention. It felt like swift took it to the next level, so far as my introductory tutorial states some of the properties of an instance can be optional type as well, that is being represented with a question mark in the end, like String?, that is just a special string. Optional actually can be used for chained calls and if it fails to reach to the end of the expression it returns nil. So it saves us from writing a lot of lets and if clauses.

let x = display?.text.hasValues{}

point to note that, x will be an optional in that case because we can never get a value return from this kind of chained methods.

We even can define this optional strings in our code.

var label: String?;

Its interesting and completely make sense that for getting back data from dictionaries we get an optional type in return. Also AnyObject is a type that takes anything and everything through a parameter or pretty much anywhere as a type but we might want to use as to treat that anyobject to something else let foo=ao as? String{}. as is just casting it to another class.

These special we actually need to take special care when handling with these optional properties. We need to unwrap that variable with a bang(!).

func touchButton(button: UIButton){
 let text = button.currentTitle!
 print("pressed \(text) button")
}

I have mentioned earlier that you can’t define a variable when you do not initialize it. Let me add more information to that statement, we actually can implicitly unwrap a variable while defining it as well, that will reduce our burden of unwrapping it every time we intended to use it.

var display: UILabel!
print("\(display.text)")

var cash: Double {
set: {
  return Double(label.text!)
}
get:{
  label.text = cash
}
}

Apart from computed properties, there are actually few other interestng properties too, like observer properties. willset and didset could be one of them.

Swift function declaration could be very interesting as well, we can define internal and external parameters in swift functions, internal parameter has scope locally when external parameters are the parameter the callee of the method has to fullfil.

Like python it provides options for positioned parameter, in swift it can be done using an underscore before the parameter name, which I did not like most, but it is how they speak swift.

Java has some support for lambda expressions, python and ruby has way better support for it though. My first impression on swift anonymous methods was pretty interesting, I would say, it is even easier.

{str1 :String, stt2:String} ~> String in
return str1 + str2

Swift has options for anonymous abstract methods as well, which can be written as following:

(Double, Double)~> Double

Need to mention that in like python or ruby functions are also considers as object in swift. So does this previous implementation of abstract class. I will be talking about its use cases in a while.

Speaking of enums I am not a great fan of python or ruby enums so I would deliberately avoid talking about them, Java has great support for enums, it supports internal methods in enum which is great, I am glad that swift also supports internal methods like java even more interesting way actually. I guess in java I did not have the credibility to use separate initializer for separate cases, what swif offers,

enum MathOperator{
 case None
 case Constant(Double)
 case Unary((Double)~> Double)
 case Binary((Double, Double)~> Double)
}

As you can see, now we have options to define our enums with so many informations, even with functions.

We can have a dictionary of our operatons like following in swift:

let operations: Dictionary<String, MathOperator> ={
 "e": MathOperator.Constant(M_E),
 "±": MathOperator.Unary(sign_change),
 "+": MathOperator.Binary({op1 :Double, op2:Double} ~> String in
  return str1 + str2
 )
}
func sign_change(number: Double) ~> Double{
 return -1* number
}

Now when we need to use this we can use swift switch statements, point to note on swift switches are they are not fall through like typical switches, if we want it then we would need fallthrough to make it fall to next case:

func sign_change(number: Double) ~> Double{

 if let op = operations["+"]{
   switch op:
    case .Constant (let value):
      return value
    case .Unary (let function):
      return function(input1)
    case .Binary(let function):
      return function(input1,input2)
 }
}

Java does not support struct, but as far as I know there are ways to implement struct like objects in java, I have heard people saying that. I have read, and seen people bundling properties in struct in ruby and python, I have used struct for classes for temporary uses and data bundling in ruby. In swift I think it has huge scope for using it for temporary classes, I have been warned that every time I am creating a class it is possibly taking a whole new space in the memory unlike a class reference. So I need to be a little bit warned when using structs, but swift has its own beautiful way to manage it, like every time only when the data changes in swift struct it consumes the memory and it has intelligence to figure out which part of data it need to reallocate, not all of them always. Enums or structs stores its properties as a constant so we can’t change it without recreating the struct in the heap and there are special ways to do that using function like putting mutating func while declaring a function.

I really liked the way I can ensure type order in swift touples, I missed this sort of declaration while working in python as I wanted to have somewhat a gurantee that a function return type will be somewhat fixed.

let x: (String, Int, Double) = {"hello world", -1, 0.2}

Recently i had a great fight with python ands its unicode to make a work done perfectly. We need to add special comments at the header of script. Swift was a lot programmer friendly in that regard, Swift unicodes are full unicodes.

I guess sometime I will have to use objective C codes, as the tutor mentioned about NSObject class. To use some of the objective c apis I would need to inherit NSObject class. It has many other NS objects, I don’t think it would be very interesting thing to talk about. So deliberately ignoring them.

CentOS postgis setup: You need JSON-C for ST_GeomFromGeoJSON

I am actually struggling with this error on our centos server all day, this is how I fixed it. I could not recover everything from my commandline history but this is what I could find out. Hope it will be helpful. I am sure while cleaning up my server for production it will be helpful to me. So cheers!

All this started with this error message on my centos server:

npm-2 Unhandled rejection SequelizeDatabaseError: You need JSON-C for ST_GeomFromGeoJSON
npm-2     at Query.formatError (/home/centos/jobcue.com/node_modules/sequelize/lib/dialects/postgres/query.js:357:14)
npm-2     at null. (/home/centos/jobcue.com/node_modules/sequelize/lib/dialects/postgres/query.js:88:19)
npm-2     at emitOne (events.js:77:13)
npm-2     at emit (events.js:169:7)
npm-2     at Query.handleError (/home/centos/jobcue.com/node_modules/pg/lib/query.js:108:8)
npm-2     at null. (/home/centos/jobcue.com/node_modules/pg/lib/client.js:171:26)
npm-2     at emitOne (events.js:77:13)
npm-2     at emit (events.js:169:7)
npm-2     at Socket. (/home/centos/jobcue.com/node_modules/pg/lib/connection.js:109:12)
npm-2     at emitOne (events.js:77:13)
npm-2     at Socket.emit (events.js:169:7)
npm-2     at readableAddChunk (_stream_readable.js:153:18)
npm-2     at Socket.Readable.push (_stream_readable.js:111:10)
npm-2     at TCP.onread (net.js:531:20)

When I tried to install json-c on server it was like:

sudo yum install json-c
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.eecs.wsu.edu
 * epel: s3-mirror-us-west-2.fedoraproject.org
 * extras: linux.mirrors.es.net
 * updates: mirror.raystedman.net
Package json-c-0.11-4.el7_0.x86_64 already installed and latest version
Nothing to do

Then I started panicing. Tried 5-6 hours long yum battles and figure out a solution that would look like following:

Install some dependencies at first:

yum install geos-devel.x86_64
yum install proj-devel.x86_64
yum install gdal-devel.x86_64
yum install libxml2-devel.x86_64
yum install json-c-devel.x86_64

yum install postgresql92-devel
sudo yum install postgresql-server 

sudo yum install geos geos-devel
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
gzip -d proj-4.8.0.tar.gz
tar -xvf proj-4.8.0.tar
cd proj-4.8.0
./configure
make
sudo make install

I needed to install gdal:

installing gdal:

sudo rpm -Uvh http://elgis.argeo.org/repos/6/elgis-release-6-6_0.noarch.rpm
sudo yum install -y gdal
./configure
make
make install

Obviously I needed to install json c:

sudo yum install json-c-devel

I needed to know where it is located:

rpm -ql json-c json-c-devel

for me it was at:

/usr/include/*

Now it is time to built our postgis like this:

wget http://download.osgeo.org/postgis/source/postgis-2.2.1.tar.gz
tar xvzf postgis-2.2.1.tar.gz
cd postgis-2.2.1
./configure --with-jsonc=/usr/include

make
make install
sudo make install

In app user to user chat using auth0 and firebase

So we are using auth0 for our user authentication, we have a server of our own which runs using relational database which seems to be perfect for our use case. Now we need to install a chat application for our user to user communication. How do we do that? Chat is a big data issue, and we need faster communication that happens in real time. How do we do that? We are using firebase for this. This is actually first time I am using firebase, to be more precise its my second day with firebase so I am quite excited that I am writing about this cool technology. The goal of this post is not only to help other but also to get help from experienced. Please point out my mistake and tell me how can I improve this.

Firebase supports actually social logins pretty well, but since we already has relied on auth0 for authentication, we won’t bother firebase to handle that. Firebase supports custom authentication, we will be using that. To issue a custom authentication token we actually need to use our firebase secret, we can’t let any user to see that so ideally which means our API server need to issue that then other client will use. Auth0 actually made our life a lot easier, we can add a firebase plugin to auth0 and auth0 will generate that custom authentication token for us for now on, so we don’t need to do that using api anymore.

I appreciate their initiative for firebase but there was some limitation at their service as well, they actually do not provide an authentication uid straight away for firebase rule auth.uid which is a must have before we start writing our firebase rules. But don’t panic we actually have a solution for that. We solved this problem by adding a simple auth0 rule:

function (user, context, callback) {
  if (context.protocol === 'delegation') {
    return  callback(null, user, context);
  }

  user.firebase_data = {
    uid: ""+user.user_id,
    user_id: ""+user.user_id
  };
  
  callback(null, user, context);
}

Now we are good to write our rules for firebase. We understand firebase is completely schema less but we will chalk down pseudo schema, a model for our database.

I think the key idea behind a beautiful designing is to see what is coming next, we want to design our model in a way that it becomes easier for it to its evolution. Later at some point we want to use our app for a conference chat, we want to let user edit and delete their messages. We want to avoid any sort of data duplication anywhere possible.

So we will have a conversations that each of the participants can access upon permission.

Lets do the relatively hard stuff first, we will design our conversation model. Roughly a conversation will have a limited array of members and lots and lots of message entries. So it should look like following:

conversations:{
 conversation0:{
  members:{
    0: user0,
    1: user1,
    ...
  }
  messages:{
    messages0:{
      message: "lets chat!",
      fromUserId: user0,
      createdAt:  timestamp
    }
  }
}

But how are our users going to access their rooms? Yet we don’t have any way to figure out which conversations a user has subscribed. We also need to figure out which conversation has unread messages, this is not symmetric, remember? So we would like to keep them inside our user data.

users:{
 users0:{
  conversations:{
   converation0:{
    last_read: timestamp,
    subject: weneedspace,
    added_by: timestamp,
    added_at: timestamp
   }
  }
 }
} 

Now we have everything, but are we secure enough from the bad guys out there? NOPE. We need rules based on which firebase will be protecting us.

Lets go for our users rules. Firebase apps are actually easily will go fragile if we don’t take enough care of it, so we know that user will have 4 properties at least last_read, subject,added_by, added_at.

newData.hasChildren(['last_read','subject','added_by','added_at'])

So we will be validate that they exists in the first place. Now we have the valid structure, but we actually don’t know that added_by is valid or not.

newData.child('added_by').val() = auth.uid

Let it be always the authenticated user uid. The user is authenticated but does he allowed to make someone else subscribe to that conversation? We will look into conversation membership list to validate that.

root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)

Now who has write permissions? As long as you are authenticated you are fine.

auth != null

Who can see my subscription? Privacy issue here. I won’t let ‘A’ authenticated user read my conversations. He must be ‘THE’ authenticated user.

auth != null && auth.uid == $user_id

For now we will only users to see through /users/conversations/ and write through /users/conversations/conv0/

so user rules go like this:

    "users":{
        "$user_id":{
          "conversations":{
              ".read": "auth != null && auth.uid == $user_id",
              "$conversation_name":{
                ".write": "auth != null",
                ".validate": "newData.hasChildren(['last_read','subject','added_by','added_at']) 
                              && newData.child('added_by').val() = auth.uid
                              && root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)
                              "
              },
              
            }
          }
      }

You can see that our validation relies on the existance of /conversations/conv0/members/0. So we need to ensure that we do first. While creating a conversation you must include yourself in a conversation thats how we validate.

newData.hasChild(auth.uid)

Any very simple to have write access you need to be authenticated.

auth != null

Now what would be our basic message look like?

newData.hasChildren(['fromUserId', 'message', 'sent_at'])

We also need to check if fromUserId is valid for that user or not.

auth.uid == newData.child('fromUserId').val()

Who can write messages? only if you are a member.

root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)

We have messages, and members both of them has at least someone who can write them. But we don’t have anyone who can read them. You can only read messages only if you are a member of it. We will add this to the parent of both of them.

data.child('members').hasChild(''+auth.uid)

Altogether conversations rule:

       "conversations":{
          "$conversation_name":{

                ".read": "data.child('members').hasChild(''+auth.uid)",
                "messages":{
                    "$hash": { 
                      ".write": "root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)",
                      //".validate": "auth.uid == newData.child('fromUserId').val()"
                      ".validate": "newData.hasChildren(['fromUserId', 'message', 'sent_at'])
                                   && auth.uid == newData.child('fromUserId').val()"
                    }
                  },
                "members":{
                   ".write": "auth != null", //"!data.exists()",
                   ".validate": "newData.hasChild(auth.uid)"
                  }
                
              
            }
        }

Altogather the rules are:

{
  "rules": {
 
    
    "users":{
      
        "$user_id":{
          "conversations":{
            
              
              ".read": "auth != null && auth.uid == $user_id",
              "$hash":{
                ".write": "auth != null"
              }
              
            }
          }
      },
      
      "conversations":{
          "$conversation_name":{

                ".read": "root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)",  
                ".write": "root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)",
                "messages":{
                    "$hash": { 
                      ".write": "root.child('conversations').child($conversation_name).child('members').hasChild(''+auth.uid)"
                      //".validate": "auth.uid == newData.child('fromUserId').val()"
                    }
                  },
                "members":{
                   ".write": "true", //"!data.exists()",
                   ".validate": "newData.hasChild(auth.uid)"
                  }
                
              
            }
        }
  }
}

Now we will add a quick proof of concept demo chat app using js and html.

Lets do auth0 sign in first:

signin.html

<script src="http://cdn.auth0.com/w2/auth0-6.7.js"></script>
<script
 src="https://code.jquery.com/jquery-1.12.3.min.js"
 integrity="sha256-aaODHAgvwQW1bFOGXMeX+pC4PZIPsvn2h1sArYOhgXQ="
 crossorigin="anonymous"></script>

<script src='https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js'></script>


<script type="text/javascript">
 var auth0 = new Auth0({
 domain: '<your auth0 >.auth0.com',
 clientID: '<your client id>',
 callbackURL: '/signin.html',
 callbackOnLocationHash: true
 });


 auth0.login({ popup: true }, function(err, profile, id_token, access_token, state, refresh_token) {
 if (err) {
 // Handle the error!
 alert(err)
 return;
 }
 $.cookie("auth0token", id_token);

var options = {
 id_token: id_token, // The id_token you have now
 api: 'firebase', // The type of app (Auth0 can generate multiple token formats)
 scope: "openid name email" // default: openid. Add any other scopes you need, i.e. "openid {scope1} {scope2}"
 };

auth0.getDelegationToken(options, function (err, delegationResult) {
 // Call your API using delegationResult.id_token
 console.log(err)
 $.cookie("firebasetoken", delegationResult.id_token);

$(location).attr('href', 'minchat.html')


 });


 // refresh_token is sent only if offline_access is set as a scope
 });

</script>
</pre>

Now we will write our client:

minichat.html
<pre>
<html><head>
 <meta http-equiv="content-type" content="text/html; charset=UTF-8">
 <meta name="robots" content="noindex, nofollow">
 <meta name="googlebot" content="noindex, nofollow">

<script type="text/javascript" src="//code.jquery.com/jquery-1.11.0.js"></script>
 <link rel="stylesheet" type="text/css" href="/css/result-light.css">
 <link rel="stylesheet" type="text/css" href="//netdna.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap.min.css">
 
 
 
 <script type="text/javascript" src="//netdna.bootstrapcdn.com/bootstrap/3.3.0/js/bootstrap.min.js"></script>
 <script type="text/javascript" src="https://cdn.firebase.com/js/client/2.2.1/firebase.js"></script>

<script src='https://cdnjs.cloudflare.com/ajax/libs/jquery-cookie/1.4.1/jquery.cookie.min.js'></script>
 


<style type="text/css">
 .tabs-left {
 border-bottom: none;
 padding-top: 2px;
 }

.tabs-left {
 border-right: 1px solid #ddd;
 }

.tabs-left>li {
 float: none;
 margin-bottom: 2px;
 }

.tabs-left>li {
 margin-right: -1px;
 }

.tabs-left>li.active>a,
 .tabs-left>li.active>a:hover,
 .tabs-left>li.active>a:focus {
 border-bottom-color: #ddd;
 border-right-color: transparent;
 }

.tabs-left>li>a {
 border-radius: 4px 0 0 4px;
 margin-right: 0;
 display: block;
 }
 .bubble{
 width: 74%;
 padding: 4px;
 border-radius: 4px;
 }
 .you{
 background-color: skyblue;
 float: left;
 text-align:left;
 }
 .me{
 background-color: orange;
 float: right;
 text-align: right;
 }

</style>

<title>In app Chat</title>


<script type="text/javascript">//<![CDATA[
$(window).load(function(){

var FirebaseChat = (function(){
 function FirebaseChat() {
 this.firebase = new Firebase('https://jobcue.firebaseio.com/');
 }
 var instance;
 return {
 get: function(){
 if (instance == null) {
 instance = new FirebaseChat()
 instance.uid = instance.firebase.getAuth().uid

var token = String($.cookie("firebasetoken") );

instance.firebase.authWithCustomToken(token, function(error, result) {
 if (error) {
 console.log("Authentication Failed!", error);
 } else {
 console.log("Authenticated successfully with payload:", result.auth);

console.log("Auth expires at:", new Date(result.expires * 1000));

// alert( JSON.stringify (firebase.getAuth()) )
 }
 });


 // Hide the constructor so the returned objected can't be new'd...
 instance.constructor = null;
 }
 return instance;
 },
 uid: function(){
 return get().getAuth().uid;
 }
 };
})();

function User(id){
 this.id = id;

this.firebaseRef = FirebaseChat.get().firebase.child('users').child(this.id) 
}

User.prototype.subscribe = function(conversation_name){
 
 c={}
 c.subject = conversation_name
 c.last_read = Firebase.ServerValue.TIMESTAMP
 c.added_by = FirebaseChat.get().uid
 c.added_at = Firebase.ServerValue.TIMESTAMP
 alert(conversation_name)
 this.firebaseRef.child('conversations').child(conversation_name).set(c)

};

User.prototype.getId = function(){return this.id };

User.prototype.read = function(conversation_name){
 c={}
 //c.subject = conversation_name


this.firebaseRef.child('conversations').child(conversation_name).child('last_read').set(Firebase.ServerValue.TIMESTAMP) 
};
User.prototype.subscriptions = function(){
 this.firebaseRef.child('conversations') 
};


User.prototype.onNewSubscription = function(callback){

this.firebaseRef.child('conversations').on('child_added', 
 function(conversation_snap){

callback( conversation_snap.key() ) 
 })

};

function Conversation(name){
 this.name = name
 this.firebaseRef = FirebaseChat.get().firebase.child('conversations').child(this.name)
}

Conversation.prototype.create = function(participants){
 firebaseRef = this.firebaseRef.child('members')

participants.forEach(function(user){

var u={}
 u.added_at = Firebase.ServerValue.TIMESTAMP
 u.added_by = FirebaseChat.get().uid

firebaseRef.child( user.getId() ).set(u) 
 })

};


Conversation.prototype.sendMessage = function(message){
 var m = {}
 m.message = message
 m.sent_at = Firebase.ServerValue.TIMESTAMP
 m.fromUserId = FirebaseChat.get().uid

this.firebaseRef.child('messages').push(m)
}


Conversation.prototype.onNewMessage = function(callback){
 this.firebaseRef.child('messages').on('child_added', 
 function(new_message_snap){ 
 callback(new_message_snap.val() ) 
 } )
}


var me = new User( FirebaseChat.get().uid );

me.onNewSubscription(function(new_conversation){
 addPeople(new_conversation)

var conversation = new Conversation(new_conversation)
 conversation.onNewMessage(function(new_message){
 addMessage(new_conversation, new_message)

})
})


$('#messageInput').keypress(function(e) {
 if (e.keyCode == 13) {


 var conversation = new Conversation($('.active').find('.name').text())

conversation.sendMessage($('#messageInput').val())

}
});

var n = 7;

function addPeople(conversation_list) {
 $("#conversations").append(
 '<li ><a href="#' + conversation_list + '" data-toggle="tab"><span class="name">' + conversation_list + '</span></a></li>'
 );

$("#individual_conversations").append(
 '<div class="tab-pane" id="'+conversation_list+'"><div style="width:100%">'+conversation_list+'</div></div>'
/* 
 '<div class="chat" data-chat="person' + (n++) + '" id="' + conversation_list.key() +
 '"><div class="conversation-start"></div>'
*/
 );
 bind();
}

function addMessage(conversation_list, msg) {
 $("#" + conversation_list).append(

'<div class="bubble ' + (msg.fromUserId == FirebaseChat.get().uid ? 'you' : 'me') + '"><span><b>'+
 (msg.fromUserId == FirebaseChat.get().uid ? 'you: ' : '')
 +'</b>' +
 msg.message+
 '</span><b>'+
 (msg.fromUserId == FirebaseChat.get().uid ? '' : ' :'+(msg.fromUserId))+
 '</b></div>'


 );


}

function bind(){
$("#conversations li a").on("click", function(){
 var conversation =$(this).find(".name").text(); 
 if(me !=null){
 me.read(conversation)
 } 
})
}


$("#newConv").on("click", function(){
 var c = $("#newConversationInput").val();
 var u = $("#toUserInput").val();
 var m =$("#newMessageInput").val();

if(me !=null){
 
 var auser = new User(u);

var con = new Conversation(c)
 con.create( [me, auser] )
 con.sendMessage(m);

me.subscribe(c);
 
 auser.subscribe(c);


 } 
})

});//]]>


</script>


</head>

<body>

<div class="col-sm-6">
 <div class="row">
 <h3>Messages
 <!-- Button trigger modal -->
 <button class="btn btn-primary btn-sm pull-right" data-toggle="modal" data-target="#myModalNorm">
 New Message
 </button>

</h3>

<hr>
 <div class="col-xs-4">
 <!-- required for floating -->
 <!-- Nav tabs -->
 <ul class="nav nav-tabs tabs-left sideways" id="conversations">
 <li class="center">Discussion </li>
 <!--li class="active"><a href="#home-v" data-toggle="tab">Home</a></li-->
 </ul>

</div>

<div class="col-xs-8">
 <!-- Tab panes -->
 <div class="tab-content" id="individual_conversations">

<!--div class="tab-pane active" id="home-v">Home Tab.</div!-->
 </div>


 </div>
 </div>


<div class="row">
 <div class="col-xs-4">SEND MESSAGE:</div>
 <input class="col-xs-8" id="messageInput" placeholder="write message and press ENTER">
</div>



</div></body></html>

<!-- Modal -->
<div class="modal fade" id="myModalNorm" tabindex="-1" role="dialog" aria-labelledby="myModalLabel" aria-hidden="true">
 <div class="modal-dialog">
 <div class="modal-content">
 <!-- Modal Header -->
 <div class="modal-header">

<h4 class="modal-title" id="myModalLabel">
 New Message
 </h4>
 </div>

<!-- Modal Body -->
 <div class="modal-body">


 <div class="form-group">
 <input class="form-control" type='text' id='newConversationInput' placeholder='Conversation'>

<input class="form-control" type='text' id='toUserInput' placeholder='to'>
 <textarea class="form-control" type='text' id='newMessageInput' placeholder='Message'></textarea>

<button id="newConv" type="submit" class="btn btn-primary">Submit</button>

<button type="button" class="btn btn-primary" data-dismiss="modal">
 Close
 </button>
 
 </div>


 </div>
 </div>
 </div>

So we are ready to go. go to and it will redirect to after signing in.

setup nodejs api server powered by passport auth0 strategy

We are using auth0 for our nodejs project, I would say it has made our life significantly better. Now we have considerably less burden on authentication process. As probably I was new in node js world it took me quite a time to figure out how to actually collect authentification token from auth0 mainly because they are using quite different term for their token than what I am used to, and I could not find a good straight forward tutorial on how to do that (I am not a great reader, I just read their example codes and it was not there). So here is how I did it:

In my setup-passport.js file I have got following:

var passport = require('passport');
var Auth0Strategy = require('passport-auth0');

    var models = require('./models/index');


var strategy = new Auth0Strategy({
    domain:       'x.auth0.com',
    clientID:     'xxx',
    clientSecret: 'xxxxxxx',
    callbackURL:  'http://localhost:3000/callback'
  }, function(accessToken, refreshToken, extraParams, profile, done) {
    // accessToken is the token to call Auth0 API (not needed in the most cases)
    // extraParams.id_token has the JSON Web Token
    // profile has all the information from the user

    //may like to create new user here;
    console.log({extra_params: extraParams});

);

passport.use(strategy);

// This is not a best practice, but we want to keep things simple for now
passport.serializeUser(function(user, done) {
  done(null, user);
});

passport.deserializeUser(function(user, done) {
  done(null, user);
});

module.exports = strategy;

And I have this at my app.js I have added following:

var passport = require('passport');

// This is the file we created in step 2.
// This will configure Passport to use Auth0
var strategy = require('./setup-passport');

// Session and cookies middlewares to keep user logged in
var cookieParser = require('cookie-parser');
var session = require('express-session');



app.use(cookieParser());
// See express session docs for information on the options: https://github.com/expressjs/session
app.use(session({ secret: 'YOUR_SECRET_HERE', resave: false,  saveUninitialized: false }));
//...
app.use(passport.initialize());
app.use(passport.session());


/*
// Auth0 callback handler
app.get('/callback',
passport.authenticate('auth0', { failureRedirect: '/url-if-something-fails' }),
function(req, res) {
if (!req.user) {
throw new Error('user null');
}
res.send({token: req.user});
//res.redirect("/user");
});
*/

app.get('/callback', function(req, res, next ){
passport.authenticate('auth0', function(err, user, info) {
if (err) { return next(err) }

return res.json( { credentials: info, user: user })

})(req, res, next);
});

My auth.js looks like:

var dotenv = require('dotenv');
dotenv.load();

var jwt = require('express-jwt');

module.exports =  jwt({
    secret: new Buffer(process.env.AUTH0_CLIENT_SECRET, 'base64'),
    audience: process.env.AUTH0_CLIENT_ID
  });

Routes that needs authentication looks like this:

var express = require('express');
var router = express.Router();
var model = require('../models/index');


var authenticate = require("../auth")

/* GET users listing. */
router.get('/', authenticate, function(request, response, next) {

    model.User.find({
        where:{ 
            providerId: request.user.sub,
         },

    }).then(function(user) {        
        response.send(user);
    });  
});

Here I go, I have got a functional authentication using passport auth0 strategy.

The “id_token” that we get from /callback is our authentication token.

$ token = "your id_token from /callback" 
$ curl -v -X GET  -H "Content-Type: application/json" -d '{}' http://127.0.0.1:3000/users -H "Authorization: Bearer $token"

Install PostGIS on your ubuntu 15.10

It took me quite a lot of time to figure out how do you do that! Though it looks like it is just 3 apt get install command, so simple right? but took quite a toll of my life to figure those 3 out. If you are reading this, probably you are also going through something semilar!

>sudo apt-get install postgresql postgresql-contrib postgis
> psql --version
psql (PostgreSQL) 9.5.2

> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
ERROR: could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory

>sudo apt-get install postgresql-9.5-postgis-2.0

> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
ERROR: could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory

> sudo apt-get install postgresql-9.5-postgis-scripts
> sudo psql --username=postgres --dbname=jobcue -c "CREATE EXTENSION postgis;"
Password for user postgres:
CREATE EXTENSION

[Code snippet] Django rest framework social oauth2 user sign up and sign in

At my serializers.py I have got the following:

from django.contrib.auth.models import User
from rest_framework import serializers
 
 
class UserSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = ('username', 'email', 'password')

    def create(self, validated_data):
        return User.objects.create_user(**validated_data)

at my views.py I have got the following:

import serializers
from rest_framework.decorators import permission_classes, api_view
from rest_framework.permissions import AllowAny
from django.views.decorators.csrf import csrf_exempt
from rest_framework.parsers import JSONParser
from rest_framework.response import Response
from rest_framework import status
import json
from django.http import JsonResponse

from oauth2_provider.models import Application, AccessToken

@permission_classes((AllowAny,))
@csrf_exempt
@api_view(['POST'])
def create_auth(request, format=None):
    if request.user.is_authenticated():
        return Response({"already_registered": "User with that username has already registered"}, status=701)
    
    data = request.data
    print data

    
    serializer = UserSerializer(data=data, partial=True)
    if serializer.is_valid():
        u=serializer.save(username= data.get(u'username') )
        application=Application.objects.create(user=u, client_type="public", authorization_grant_type="password",name="general")
        client_id = application.client_id #call the url to get your tokens, use urllib or something similar
        client_secret = application.client_secret
        return JsonResponse({'client_id': client_id, 'client_password' : client_secret}, status=201)
    else:
        return JsonResponse({'errors': serializer.errors}, status=400)

I have added following at my urls.py

urlpatterns = patterns(
    '',
    url(r'^register/', 'social_app.views.create_auth'),
    url(r'^auth/', include('rest_framework_social_oauth2.urls')),
)

Testing:

 
sadaf2605@pavilion-g4:~$ curl -X POST -H "Content-Type: application/json" -d '{"email":"boba@athingy09876.com", "password":"p4ssword", "username": "user100"}' http://localhost:8000/register/

returns:

{"client_password": "EjQKMCAGmsUEm3L26uO7XSKnrZZVSVBQJUuvqfwi63pRB7d5y3ndlbZV0cBgQU7t3lCy078DS0FLqhaYoe9JZF0cQCIAgFKo7lfYU3npP7Eyv1PLk2eLPRnD3lF3OUUP", "client_id": "JhbwqqvE34vVjWiuMPnkV1eE636QQ3SzyQXLjmgs"}
sadaf2605@pavilion-g4:~$ curl -X POST -d "client_id=JhbwqqvE34vVjWiuMPnkV1eE636QQ3SzyQXLjmgs&client;_secret=EjQKMCAGmsUEm3L26uO7XSKnrZZVSVBQJUuvqfwi63pRB7d5y3ndlbZV0cBgQU7t3lCy078DS0FLqhaYoe9JZF0cQCIAgFKo7lfYU3npP7Eyv1PLk2eLPRnD3lF3OUUP&grant;_type=password&username;=user100&password;=p4ssword" http://localhost:8000/auth/token
{"access_token": "bssEYlDNaXefq8TPNRuu8oLolqYJp2", "token_type": "Bearer", "expires_in": 36000, "refresh_token": "fankCVPC3P84pQWI5oWOIhtWLCky4w", "scope": "read write"}