Author Archives: Ruihong Huang

Read Parquet in R

The new LOBSTER engine working on AWS cloud, can output order book data in either Parquet or CSV format . For the sake of efficiency, it outputs Parquet file by default. For Spark and Python users, loading the parquet files … Continue reading

Posted in R | Leave a comment

Clawing in the Jungle

Arnaud Amsellem just publishes an exciting research  using LOBSTER data.  – Using random forest to model limit order book dynamic

Posted in LOBSTER Data | Leave a comment

How to Install Cygwin without Admin Privilege

Download Cygwin setup file here. Right click on setup-x86_64.exe Modify the shotcut property, adding " –no-admin" at the end of the target. Note that there must have a space between ".exe" and "–no-admin". Double click the shortcut to run the … Continue reading

Posted in Cygwin | Leave a comment

MathWorks Publishes Machine Learning Application Using LOBSTER Data

Posted in LOBSTER Data | Leave a comment

Self Type in Scala

abstract class A (val x: Int){ } trait B { def xPlus1: Int = x +1 } Of course this is silly. Because B does not know x defined in A, Scala compiler will complain, error: not found: value x … Continue reading

Posted in Scala | Leave a comment

Mount AWS S3 Bucket on An EC2 Ubuntu Instance

The current Ubuntu version on AWS EC2 instances is 18.04 LTS. After launch an Ubuntu Instance and connect to it by SSH, one can follow those steps to set up a permanent mount to S3 bucket. Install the packages. $ … Continue reading

Posted in AWS | Leave a comment

Convert A Spark DataFrame with Date Columns to Pandas DataFrame using Apache Arrow

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. After install pyarrow package, we can convert Spark DataFrame … Continue reading

Posted in Python, Spark | Leave a comment

Tips on Linux

How to unzip all 7z files in a directory? One can use the following bash script to unzip all *.7z files in a directory. #!/bin/bash FILES=$1/*.7z shopt -s nullglob for f in $FILES do echo "Processing $f file …" 7za … Continue reading

Posted in Linux | Leave a comment

Evaluate Trading Strategies by Using LOBSTER Data

— A short review of a working paper by Balch (2019) from J.P. Morgan Artificial Intelligence Research and Imperial College London T.H. Balch have published a working paper, “How to Evaluate Trading Strategies: Single Agent Market Replay or … Continue reading

Posted in LOBSTER Data | Leave a comment

LOBSTER on Spark

A new LOBSTER engine has been built on Spark. The demo Jupyter-notebook is available at here .

Posted in LOBSTER Data, Python, Spark | Leave a comment