Getting valid time spans from timestamped data using Python Pandas.

Assume we have a Panda's DataFrame indexed by a time.
2017-12-15 00:00:00.....
2017-12-15 00:00:15.....
2017-12-15 00:00:29.....
2017-12-15 00:00:46.....
2017-12-15 03:31:01.....
2017-12-15 03:31:17.....
2017-12-15 03:31:29.....
2017-12-15 03:32:16.....
And if we want to get time spans from the data under condition that if rows' intervals is less than threashold, make it one span.
startend
2017-12-15 00:00:002017-12-15 00:00:46
2017-12-15 03:31:012017-12-15 03:32:16
you can do it by 
import pandas as pd
from datetime import timedelta

threashold = timedelta(seconds=30)

df = pd.read_csv("original_data.csv")
intervals = df_origin.index[1:] - df_origin.index[:-1]
df[HEAD][1:] = intervals > threshold
df[TAIL][:-1] = intervals > threashold
df[HEAD][0] = True
df[TAIL][-1] = True

_df = df[df["HEAD"] ^ df["TAIL"]]

result = pd.DataFrame({"start":_df[_df["HEAD"]].index, "end":_df[_df["TAIL"]].index}, columns=["start","end"])
Point is to mark rows after long interval "Head" and rows before long interval "TAIL", then remove both false (interval data) or both true (isolated data).

Comments

Post a Comment

Popular posts from this blog

Calling OpenCV functions via Cython from Python 3.X.

I have managed to run OpenCV function from Python3 via Cython today, so I want to write about it. Before you start something like this, you should know you have to read at least header files of C library to write right definition in Cython,because Cython does not do that for you. I am using virtualenv, python from pythonz, and home-brewed OpenCV. Almost everything of C++ works just fine with Cython, but still, you should do some Voodoo things like for integer template argument. ctypedef void* int_parameter ctypedef int_parameter two "2" ctypedef Point_[float, two] Point2f Now, main sample Cython code is following. #===== testopencv.pyx ======== import numpy as np cimport numpy as np # for np.ndarray from libcpp.string cimport string from libc.string cimport memcpy cdef extern from "opencv2/core/core.hpp": cdef int CV_WINDOW_AUTOSIZE cdef int CV_8UC3 cdef extern from "opencv2/core/core.hpp" namespace "cv": cdef cppclass Mat: Ma...
Read more

Showing CPU/Memory usage on tmux status bar(tmuxのステータスバーにCPUとMemoryの使用状況を表示する)

Image
On OSX, to show CPU usage, script is like this. #!/bin/bash top -l 1 | head -n 4 | tail -n 1|awk '{printf "%3d%%\n", $3}' to get Memory Usage, script is like this. #!/bin/bash vm_stat | awk 'BEGIN{FS="[:]+"}{if(NR<7&&NR>1)sum+=$2; if(NR==2||NR==4||NR==5)free+=$2} END{printf "%3d%%\n",100*((sum - free)/sum)}' Name thsese scripts like getCpuUsage.sh/getMemUsage.sh and put them in folder where the $PATH is set. Change their mode to u+x. To show them on tmux status line, .tmux.conf is like set -g status-right '#[fg=blue,bold]#H#[default] #[fg=blue,bold][CPU=#(getCpuUsage.sh) MEM=#(getMemUsage.sh)]#[default]'
Read more

Subclassing and Signal connect on a same widget crashes PySide application on exit.

In pyside, I wanted to do something like these, class MyTextLine(QLineEdit): def focusInEvent(self, e): #do something QLineEdit.focusInEvent(self, e) .... widget = MyTextLine(QLineEdit) widget.editingFinished.connect(some_callback) However, this app tends to crash on exit. Core dump saids it is something about destructor of a signal manager. Details are rather vague, but I managed to avoid crash deleting signal-connect. class MyTextLine(QLineEdit): def focusInEvent(self, e): #do something QLineEdit.focusInEvent(self, e) def setCallback(self, cb): self.cb = cb def focusOutEvent(self, e): if hasattr(self, 'cb'): self.cb() .... widget = MyTextLine(QLineEdit) widget.setCallback(some_callback) If anyone knows why this happens, please teach me! One more thing, if you use QTreeWidget having widgets on it, it seems better to do clear() it before closing your application. Update 2015/01/25: It seems like widget.se...
Read more