Using bulkload cause UnicodeDecodeError (Google App Engine) GAEのbulkloadでUnicode Decode Error

I've been wondering a following loader.py causes unicode decode error when uploading utf-8 csv file by bulkloader to Google App Engine.
class A(bulkloader.Loader):
def __init__(self):
 bulkloader.Loader.__init__(self, 'TEST',
  [("__key", dummy), ("prop", lambda x: x.decode('utf-8'))])

def generate_key(self, i, values): return values[0]


I tried to avoid UnicodeDecodeError by 
x.decode('utf-8')
but it didn't work.
After seeing around datastore.py of Google App Engine code, it turns out (of course) the key also should be unicoded.
So correct code is
class A(bulkloader.Loader):
def __init__(self):
 bulkloader.Loader.__init__(self, 'TEST',
  [("__key", dummy), ("prop", lambda x: x.decode('utf-8'))])

def generate_key(self, i, values): return values[0].decode('utf-8')
It took a few hours ...

Comments

Post a Comment

Popular posts from this blog

Subclassing and Signal connect on a same widget crashes PySide application on exit.

In pyside, I wanted to do something like these, class MyTextLine(QLineEdit): def focusInEvent(self, e): #do something QLineEdit.focusInEvent(self, e) .... widget = MyTextLine(QLineEdit) widget.editingFinished.connect(some_callback) However, this app tends to crash on exit. Core dump saids it is something about destructor of a signal manager. Details are rather vague, but I managed to avoid crash deleting signal-connect. class MyTextLine(QLineEdit): def focusInEvent(self, e): #do something QLineEdit.focusInEvent(self, e) def setCallback(self, cb): self.cb = cb def focusOutEvent(self, e): if hasattr(self, 'cb'): self.cb() .... widget = MyTextLine(QLineEdit) widget.setCallback(some_callback) If anyone knows why this happens, please teach me! One more thing, if you use QTreeWidget having widgets on it, it seems better to do clear() it before closing your application. Update 2015/01/25: It seems like widget.se
Read more

Calling OpenCV functions via Cython from Python 3.X.

I have managed to run OpenCV function from Python3 via Cython today, so I want to write about it. Before you start something like this, you should know you have to read at least header files of C library to write right definition in Cython,because Cython does not do that for you. I am using virtualenv, python from pythonz, and home-brewed OpenCV. Almost everything of C++ works just fine with Cython, but still, you should do some Voodoo things like for integer template argument. ctypedef void* int_parameter ctypedef int_parameter two "2" ctypedef Point_[float, two] Point2f Now, main sample Cython code is following. #===== testopencv.pyx ======== import numpy as np cimport numpy as np # for np.ndarray from libcpp.string cimport string from libc.string cimport memcpy cdef extern from "opencv2/core/core.hpp": cdef int CV_WINDOW_AUTOSIZE cdef int CV_8UC3 cdef extern from "opencv2/core/core.hpp" namespace "cv": cdef cppclass Mat: Ma
Read more

Showing CPU/Memory usage on tmux status bar(tmuxのステータスバーにCPUとMemoryの使用状況を表示する)

Image
On OSX, to show CPU usage, script is like this. #!/bin/bash top -l 1 | head -n 4 | tail -n 1|awk '{printf "%3d%%\n", $3}' to get Memory Usage, script is like this. #!/bin/bash vm_stat | awk 'BEGIN{FS="[:]+"}{if(NR<7&&NR>1)sum+=$2; if(NR==2||NR==4||NR==5)free+=$2} END{printf "%3d%%\n",100*((sum - free)/sum)}' Name thsese scripts like getCpuUsage.sh/getMemUsage.sh and put them in folder where the $PATH is set. Change their mode to u+x. To show them on tmux status line, .tmux.conf is like set -g status-right '#[fg=blue,bold]#H#[default] #[fg=blue,bold][CPU=#(getCpuUsage.sh) MEM=#(getMemUsage.sh)]#[default]'
Read more