Using bulkload cause UnicodeDecodeError (Google App Engine) GAEのbulkloadでUnicode Decode Error

I've been wondering a following loader.py causes unicode decode error when uploading utf-8 csv file by bulkloader to Google App Engine.
class A(bulkloader.Loader):
def __init__(self):
 bulkloader.Loader.__init__(self, 'TEST',
  [("__key", dummy), ("prop", lambda x: x.decode('utf-8'))])

def generate_key(self, i, values): return values[0]


I tried to avoid UnicodeDecodeError by 
x.decode('utf-8')
but it didn't work.
After seeing around datastore.py of Google App Engine code, it turns out (of course) the key also should be unicoded.
So correct code is
class A(bulkloader.Loader):
def __init__(self):
 bulkloader.Loader.__init__(self, 'TEST',
  [("__key", dummy), ("prop", lambda x: x.decode('utf-8'))])

def generate_key(self, i, values): return values[0].decode('utf-8')
It took a few hours ...

Comments

Post a Comment

Popular posts from this blog

Calling OpenCV functions via Cython from Python 3.X.

I have managed to run OpenCV function from Python3 via Cython today, so I want to write about it. Before you start something like this, you should know you have to read at least header files of C library to write right definition in Cython,because Cython does not do that for you. I am using virtualenv, python from pythonz, and home-brewed OpenCV. Almost everything of C++ works just fine with Cython, but still, you should do some Voodoo things like for integer template argument. ctypedef void* int_parameter ctypedef int_parameter two "2" ctypedef Point_[float, two] Point2f Now, main sample Cython code is following. #===== testopencv.pyx ======== import numpy as np cimport numpy as np # for np.ndarray from libcpp.string cimport string from libc.string cimport memcpy cdef extern from "opencv2/core/core.hpp": cdef int CV_WINDOW_AUTOSIZE cdef int CV_8UC3 cdef extern from "opencv2/core/core.hpp" namespace "cv": cdef cppclass Mat: Ma...
Read more

How to update lookup fileld of Kintone using pykintone

 When you try to update kintone record's lookup field, you will fail see the error saying you don't have enough privilege to do that. "フィールド「XXXX」の値「YYYY」が、ルックアップの参照先のフィールドにないか、またはアプリやフィールドの閲覧権限がありません。" This is because you need to have both writing access to looking-up application and reading access to looked-up application. You can include two api tokens in header for REST API, you can also do that by pykintone just connect two api tokens by comma. import pykintone app = pykintone.app('tr3xr', "30", "tokenXXX...XXX,tokenYYY....YYY") Isn't that easy?
Read more

Android's VideoView doesn't always stretch video.

Creating a full screen simple video player on Android 2.4.x(GingerBread), I've been annoyed by the problem that VideoView doesn't always stretch videos to full screen(keeping aspect ratio). Reading VideoView's code, I finally found a solution to this. Point is to call getHolder().setFixedSize in onMeasure. class StretchVideoView(context:Context, attr:AttributeSet) extends VideoView(context, attrib) { def measure(measureSpec:Int):Int = { val specMode = View.MeasureSpec.getMode(measureSpec) View.MeasureSpec.getSize(measureSpec) } override def onMeasure(widthMeasureSpec:Int, heightMeasureSpec:Int) { val (w,h) = (measure(widthMeasureSpec), measure(heightMeasureSpec)) getHolder().setFixedSize(w,h) super.onMeasure(widthMeasureSpec, heightMeasureSpec) } } By the way, the above code is written in Scala, so please write in Java if you're not Scala guy (Usually not).
Read more