Encoding Special Characters with URLLib Quote Function in Python

I was actually testing my final year project today and there was so much noise in the data, I was really frustrated by the end of it.

One of the most troublesome and difficult to figure out was urllib.quote(movie) function.

You should see movie titles people update, here are a few.

I ♥ Bollywood, Funniest movies ever

That really seemed a challenge to be sent over for an API call. We thought of stripping them off in the sentence, but there are a few French and Italian movies which always have some or the other odd character in them. I used quote() function and was getting KeyError exception.

Finally I figured it out, you have to encode it into UTF-8 so that they can be sent across. So while calling a URL, if it has any special characters in it, better encode it and sent it accross.

Example: (Google App Engine)

import urllib
from google.appengine.api import urlfetch

data = u'♥+ツ'
url = 'http://www.google.com?search='

response = urlfetch.fetch(url + urllib.quote(data.encode(encoding = 'UTF-8')))

if response.status_code == 200:
    output = response.content
    self.response.out.write(output)

I wasted almost an hour on figuring out what to do. If you ever get a KeyError when you are using URLLib.Quote, then this is the solution.

Replacing characters in a file (C)

The lesser known topic when it comes to C is FILE I/O. Even I’ve learnt a great deal in the past few days than what I’d learnt during my C classes. Today I sat down to solve the problem I’ve been facing on how to replace characters of a file in C. I searched through ‘stack-overflow’, dream in code and googled for about a half an hour without avail, but in the end the faithful GNU GCC Reference Manual helped me. It is related to my steganography project, where I try to replace the last bit of a byte and hide information using that. Let’s get down to business.

Please note that I’m talking in the context of replacing characters in a file. For example, say you want to replace all the ‘a’ characters with ‘b’.

These are the key points that led to my downfall: (even though I wiggled my way out of it!)
  • Opening a file for I/O – You need to open a file with “r+” if you are dealing with ASCII files and “rb+” if you are dealing with Binary files. The most common mistake one does when opening a file to read/write is that we forget and open the file using the “r” mode.
  • File pointer moves forward after every read or write. I had totally missed this point even though I knew it. I wasted about fifteen minutes on this!
  • Between a fgetc() and a fputc() call you need to call fflush() so that the changes you have made to take effect (in short it will transfer the buffer content to the file).

So with these things covered I’ll write an example which will help you understand the problem and the solution.

#include <stdio.h>

void read_file(FILE *fp)
{
    char c;
    fseek(fp,0,SEEK_SET)
    while(1)
    {
        c = (char) getc(fp);
        if(c == EOF)
            break;
        else
            printf("%c",c);
    }
    printf("\n\n");
}

void replace(FILE *fp, char find_c, char replace_c)
{
    char c;
    fseek(fp,0,SEEK_SET);
    while(1)
    {
        c = (char) getc(fp);
        if(c == EOF)
            break;
        else if(c == find_c)
        {
            printf("*");
            if(fseek(fp,-1,SEEK_CUR) != 0)
            {
                printf("Function: replace() :: Error: fseek() not working.\n");
                return -1;
            }
            if(fputc((int)replace_c,fp) == EOF)
            {
                printf("Function: replace() :: Error: fputc() not working.\n");
                return -1;
            }
            if(fflush(fp) == EOF)
            {
                printf("Function: replace() :: Error: fflush() not working.\n");
            }
        }
    }
    printf("\n\n");
}

int main()
{
    char find_c = 'l';
    char replace_c = 'm';
    FILE *fp;  

    printf("Before Replacement:\n");
    fp = fopen("message.txt","r");
    if(fp == NULL)
    {
        printf("Function: main() :: Error: fopen() not working for \"r\" mode.\n");
        return -1;
    }
    read_file(fp);
    fclose(fp);

    printf("During Replacement:\n");
    fp = fopen("message.txt","r+");
    if(fp == NULL)
    {
        printf("Function: main() :: Error: fopen() not working for \"r+\" mode.\n");
        return -1;
    }
    replace(fp,find_c,replace_c);
    fclose(fp);

    printf("After Replacement:\n");
    fp = fopen("message.txt","r");
    if(fp == NULL)
    {
        printf("Function: main() :: Error: fopen() not working for \"r\" mode.\n");
        return -1;
    }
    read_file(fp);
    fclose(fp);
    return 0;
}

One more thing I want to tell you is to have a copy of GCC Reference Manual with you, it really helps you a lot!