The other day I needed to scrape data from a table on a webpage. Thinking about traversing the DOM and building up an array was already giving me a headache. Thankfully pandas has the “read_html” function. Getting a list of dataframes for each table on the page was as easy as:
dfs = pd.read_html(url)pd.date_range(date_from, date_to, freq="D")
2. merge with indicator
if you set `indicator` parameter of merge() to True pandas adds a column that tells you which dataset the row came from
3. merge with approximate match - the tolerance parameter of merge_asof()
pd.merge_asof(trades, quotes, on="timestamp", by='ticker', tolerance=pd.Timedelta('10ms'), direction='backward')
4. Create an Excel report and add some charts
5. Save the dataframe in gzipped form
edit: formatting
edit2: I managed to get an outline link
Edit: not that it's a bad thing, I just mention it so people who read this know for sure they can avoid the link if they already know how to use the tips listed here. The article has code examples and more details.
EDIT: for instance, it looks like on a suitably configured environment, passing in `s3://` and `gs://` URLs works fine too...