Google App Engine: Geo location identification and reverse geo-coding example

Today, I developed an application that asks users for their location (be it PC or cell phones) and then it displays the latitude and longitude (Geo-Location identification), it even looks up the address (Reverse Geocoding) from Google Maps API. I also store HTTP headers so that this information can be used to identify the device used (I’m working on it right now!)

Okay! This might not involve a lot of App Engine Code, just some basic code to store user location (datastore) and Users API from Google.

These things which are stored can be viewed in an Administrator Area which shows the last 100 peoples’ information.

But this has a lot of jQuery and JavaScript code in it! I know jQuery sounds fancy but just visit W3CSchools jQuery tutorial and finish it (it’ll not take you more than 5-10 minutes), you’ll understand (almost) every line of code.

What I’ve used:

  • Google App Engine (webapp2, Python 2.7)
  • jQuery
  • Google Maps V3 API

The code can be found here. It’s well commented and I expect you to know a bit of GAE (templates as well), jQuery is used a lot so a bit of that as well. Lookup Google Maps documentation if you don’t understand the code written for Google Maps.

N-Gram generation in Python

UPDATE – 24/03/2018: I’m in the process of rewriting this article. For those of you who can understand a bit of non-trivial Python code you can take a look at my GitHub repository for a more elegant implementation.

OUTDATED information from here…

I’ve written a very small code snippet that actually generates n-grams. I’ve also added a small tweak that gives us the number of times a n-gram has appeared in the document.
The example I’ve considered is a Shakespeare’s play (All is Well that Ends Well). I’ll be generating the most common 3,4,5 or 6 word phrases that were used by Shakespeare in this particular play.
The first thing to do is cleaning up the document. Removing stuff like ACT1, SCENE 1, [To Derpina] etc. The next step is tokenising the document (splitting the document into tokens by stripping punctuations and white spaces).
Now we get into action:
#By now you should have a list of the words in the file
#There should not be unnecessary punctuation marks in the end
#of the words or any unnecessary white spaces as well.

#now word_list contains a list, generate a n-gram
#print word_list

#n for n-gram
#Change it to whatever the requirement is
n = 6

ngrams = dict()

#create an n-gram list
for i in range(len(word_list) - n + 1):

    gram = tuple(word_list[i:i+n])

    if gram in ngrams:
        ngrams[gram] += 1
    else:
        ngrams[gram] = 1

#now ngrams contains all the ngrams of the book
sorted_ngrams = sorted(ngrams.iteritems(), key = operator.itemgetter(1), reverse = True)

Okay! this is the only working part of this program that needs to be explained. I believe the the code is self-explanatory if you know a bit of Python.
The source code can be found in my repository .

Google App Engine: Facebook server side authentication

When you’re working with devices without JavaScript or in cases where the user has disabled JavaScript you’ll need to work out strategies to do your stuff on the server-side. A classic case is form validation using JavaScript, which fails when you don’t have a server side verification system and allows users to post junk onto your site.

I’ve just converted a small script shown in this example on Facebook’s site that does the same thing, but on PHP. This code closely emulates the one that leads to the link above.

Another case you might need this is because Facebook has scrapped its Python API support since OAuth 2.0 has been introduced.

There are some limitations to this application, one of which is as Facebook prevents POST headers, you can’t integrate this method into a canvas.
It builds upon the sessions example I’ve covered in the previous post.

The build up of the program is as follows:

There are 3 files:

  1. state_variable.py : A class that generate state variable –  a unique combination of 13/27 characters that Facebook generates when you make a request. the variable ‘code’ in the URL of a Facebook application.
  2. session_module.py: This is a class that handles the sessions. It must be inherited by any class that uses sessions. Refer to the post (the link) which covers it for more information on it.
  3. main.py: This is the main Python program that Google App Engine handles. The two step procedure to check whether the URL has the ‘code’ request variable in the request header if so continue with OAuth authentication, else redirect the user to the Facebook page on which he gives permissions/authenticates the application.

It really is a primitive code for sandboxing purposes, you’ll need to refine the code with exception handling and other stuff to actually deploy it.

Any bugs/suggestions are always welcome. I really don’t know if I’ll be working on this in any near future. I just wanted this application in public, so that if anybody needs it they can build upon this code.

The link to the code repository is here.

Google App Engine: Sessions

With the advent of Google App Engine (Python 2.7) and WebApp2, there have been many changes in the way people code on Google App Engine. WebApp2 includes a Session Management script in the module ‘webapp2_extras

This is a simple sessions example with Google App Engine with Python 2.7 and WebApp2. The code is self explanatory. I haven’t implemented exceptions and errors as I won’t be using this snippet of code anymore. But figuring out something is the fun part isn’t it?

Session Module:

#Import sessions for session handling
import webapp2
from webapp2_extras import sessions

#This is needed to configure the session secret key
#Runs first in the whole application
myconfig_dict = {}
myconfig_dict['webapp2_extras.sessions'] = {
    'secret_key': 'my-super-secret-key-somemorearbitarythingstosay',
}

#Session Handling class, gets the store, dispatches the request
class BaseSessionHandler(webapp2.RequestHandler):
    def dispatch(self):
        # Get a session store for this request.
        self.session_store = sessions.get_store(request=self.request)

        try:
            # Dispatch the request.
            webapp2.RequestHandler.dispatch(self)
        finally:
            # Save all sessions.
            self.session_store.save_sessions(self.response)

    @webapp2.cached_property
    def session(self):
        # Returns a session using the default cookie key.
        return self.session_store.get_session()
#End of BaseSessionHandler Class

Main Module:

import webapp2
from webapp2_extras import sessions
import session_module

#MainHandler class where we write code for ourselves
class MainHandler(session_module.BaseSessionHandler):
 def get(self):

  if self.session.get('counter'):
   self.response.out.write('Session is in place')
   counter = self.session.get('counter')
   self.session['counter'] = counter + 1
   self.response.out.write('Counter = ' + str(self.session.get('counter')))
  else:
   self.response.out.write('Fresh Session')
   self.session['counter'] = 1
   self.response.out.write('Counter = ' + str(self.session.get('counter')))
#End of MainHandler Class

#The application starts running after this is interpreted
app = webapp2.WSGIApplication([('/', MainHandler),], config = session_module.myconfig_dict)

Replacing characters in a file (C)

The lesser known topic when it comes to C is FILE I/O. Even I’ve learnt a great deal in the past few days than what I’d learnt during my C classes. Today I sat down to solve the problem I’ve been facing on how to replace characters of a file in C. I searched through ‘stack-overflow’, dream in code and googled for about a half an hour without avail, but in the end the faithful GNU GCC Reference Manual helped me. It is related to my steganography project, where I try to replace the last bit of a byte and hide information using that. Let’s get down to business.

Please note that I’m talking in the context of replacing characters in a file. For example, say you want to replace all the ‘a’ characters with ‘b’.

These are the key points that led to my downfall: (even though I wiggled my way out of it!)
  • Opening a file for I/O – You need to open a file with “r+” if you are dealing with ASCII files and “rb+” if you are dealing with Binary files. The most common mistake one does when opening a file to read/write is that we forget and open the file using the “r” mode.
  • File pointer moves forward after every read or write. I had totally missed this point even though I knew it. I wasted about fifteen minutes on this!
  • Between a fgetc() and a fputc() call you need to call fflush() so that the changes you have made to take effect (in short it will transfer the buffer content to the file).

So with these things covered I’ll write an example which will help you understand the problem and the solution.

#include <stdio.h>

void read_file(FILE *fp)
{
    char c;
    fseek(fp,0,SEEK_SET)
    while(1)
    {
        c = (char) getc(fp);
        if(c == EOF)
            break;
        else
            printf("%c",c);
    }
    printf("\n\n");
}

void replace(FILE *fp, char find_c, char replace_c)
{
    char c;
    fseek(fp,0,SEEK_SET);
    while(1)
    {
        c = (char) getc(fp);
        if(c == EOF)
            break;
        else if(c == find_c)
        {
            printf("*");
            if(fseek(fp,-1,SEEK_CUR) != 0)
            {
                printf("Function: replace() :: Error: fseek() not working.\n");
                return -1;
            }
            if(fputc((int)replace_c,fp) == EOF)
            {
                printf("Function: replace() :: Error: fputc() not working.\n");
                return -1;
            }
            if(fflush(fp) == EOF)
            {
                printf("Function: replace() :: Error: fflush() not working.\n");
            }
        }
    }
    printf("\n\n");
}

int main()
{
    char find_c = 'l';
    char replace_c = 'm';
    FILE *fp;  

    printf("Before Replacement:\n");
    fp = fopen("message.txt","r");
    if(fp == NULL)
    {
        printf("Function: main() :: Error: fopen() not working for \"r\" mode.\n");
        return -1;
    }
    read_file(fp);
    fclose(fp);

    printf("During Replacement:\n");
    fp = fopen("message.txt","r+");
    if(fp == NULL)
    {
        printf("Function: main() :: Error: fopen() not working for \"r+\" mode.\n");
        return -1;
    }
    replace(fp,find_c,replace_c);
    fclose(fp);

    printf("After Replacement:\n");
    fp = fopen("message.txt","r");
    if(fp == NULL)
    {
        printf("Function: main() :: Error: fopen() not working for \"r\" mode.\n");
        return -1;
    }
    read_file(fp);
    fclose(fp);
    return 0;
}

One more thing I want to tell you is to have a copy of GCC Reference Manual with you, it really helps you a lot!

Processing pixels in a Bitmap Image file

I was meddling with some image processing, I had to process some pixels from a BMP image, if you’re looking for the same then this is the blog post for you. The Algorithm is pretty simple, I’ve used the Wikipedia article BMP File format as my reference, the source is available in a GitHub repository.

//Program to analyze a Bitmap File.
//Please read the Wikipedia article follow this algorithm
//even more clearly.</pre>
//Moderate familiarity with C required to understand the
//the functions in the source code, but not the algorithm.

Start Program

Openthe BMP file

Get the BMP Magic
Print the BMP Magic
**Nothing of Importance, just
the letters 'B' and 'M'

Get the BMP Header
Print the BMP Header
**Contains the Pixel Data Offset (address)
**Other Information (Not important):
>File Size
>Creator 1
>Creator 2

Get the DIB` Header
Print the DIB Header
**Contains Width in Pixels (in pixels)
**Contains Height in Pixels (in pixels)
**Contains Pixel Depth (in bits)
**Other Information (not important):
>Header Size (in bytes)
>Compression Type (ENUM)
>Horizontal, Vertical (in pixels/m)
>No of color planes (usually 1)
>Pixel Data size (don't confuse with filesize)
>No of colors in palette (0 - default)
>No of important colors (0 - all are important)

No. of bytes per pixel = Pixel Depth / 8

Go the Pixel Data Offset
foreach x in Height (No. of rows)
{
  foreach y in Width (No. of columns)
  {
    foreach b in [No. of bytes per pixel]
    {
      print [BYTE DATA]
      //You can perform any other operation
      //once you can print the information.
    }
  }
  print NEWLINE
}

Close the BMP File
End Program

DIB: Device Independent Bitmap
NOTE: I’ve printed out all the information in the bitmap file though

The source code is available here in my GitHub repository.