izyda

Very strange what is going on w/ the URL from HN (vs. in general). Trying to figure it out, apologies

I apologize the link is down, the corrected link is here: https://cybersyn-financial.streamlit.app/

It brings together data from the following agencies:

- Federal Deposit Insurance Corporation (FDIC) - Federal Reserve Economic Data (FRED) - Federal Financial Institutions Examination Council (FFIEC) - Consumer Financial Protection Bureau (CFPB)

All of this data can quickly be visualized in the dashboard but, more importantly, can quickly be accessed from Snowflake's Marketplace (we provide the SQL queries needed to replicate our sample analysis).

Questions that can/are answered on the front page as examples include:

- Contextualizes the dramatic rise in the Fed Funds rate vs. the net interest income of banks (ie. consumer surplus capture or not!) - What is happening with deposits at Large vs. Small Banks since the collapse? - What banks are most at risk of the duration mismatch that causes the SVB run

Apologies, the fixed linked is here: https://cybersyn-financial.streamlit.app/

HN does not appear to let me fix the original link

Fixed Link: https://cybersyn-financial.streamlit.app/

I ran a data science group at a major discretionary NYC based hedge fund. We were not a traditional quant fund, but hired many software engineers.

Here is my take on what it takes: https://magis.substack.com/p/what-makes-alternative-data-sci...

Another great example of a solo-founder OSS project is HTTPie https://httpie.io/

While many know the HTTPie CLI product, they also have a desktop and web product now

Check out;

https://httpie.io/product

How about HTTPie: https://httpie.io/

Well known for the CLI, they now have a web/desktop client as well.

How does Strongbox syncing work between your mobile and desktop devices? Do you just have to store the database on your own cloud account (ie. Dropbox?)

What do you use to construct vega charts? any sort of UI-based editor?

If there were an editor available that allowed you to both have Tableau-like interactions or edit the vega spec at the same time, would you use it?

Do you build any visualizations on "live data"? (ie. backed by some updating database or API?) Or is the data generally static once you're at the visualization stage?

Do you use any GUI editors for Vega-lite (say Polestar or Voyager2)? or do you just write the specs from scratch?

I shot you (or your co-founder) a note on your Slack - let's pick it up there?

I think there is a real market need here for Business Intelligence analysts, looking for new tools along with their new datawarehouse (ie. Snowflake).

Alteryx has been around this space for forever and has had great (really astounding) success with their UI. So, I think your idea of keeping the UI but using SQL (to version to control, etc.) makes sense. In the future, it seems totally possible that you add more drag-and-drop functionality that allows non-SQL capable analysts to make queries, join things, filter, etc. (like in Alteryx) but that then actually generates SQL.

From a company building perspective, I think there is an open question whether you need to reinvent the framework yourself or use dbt under the hood (already huge community & permissive license) and simply offer a better/alternative UI targeted at a different audience.

Thanks, appreciate the clear & thoughtful answer.

What do you see as your key differentiator from Databricks? what's the key pain point they weren't/couldn't solve that you are?

Coatue Management | New York, NY | Engineers (Scala) / Data Scientists | Full-time | ONSITE | Sponsor Visas

Coatue Management is a hedge fund based in New York City. We focus on investments in the technology, media and telecommunications sectors.

We're building a market intelligence platform based on non-traditional data (ie. non-financial data). We look for unique and creative sources of data, and use it to model key firm and economic metrics.

Data Engineer: - Experience in using and tuning Spark on large datasets (>10TB) for batch processing - Preferably has experience with AWS EMR and/or has used Scala for data engineering - Bonus: Experience with other big data tech (e.g. Hadoop/YARN, Scalding, Flink) and/or lambda architectures

Keyword skills: Scala, Spark, Redshift, Lambda

Algorithmist: - Experience in Scala (or any other functional language e.g. Haskell, F#, OCaml) - Strong algorithms and data structure background e.g. competitive programming - Bonus: Prior experience in using/implementing ML/NLP algorithms at scale

Data Scientist: - We prioritize very strong skills at R (preferable!) or Python - Keen to do ad hoc data diving but also build automated tools for repeating that analysis - Employs best practices of software engineering in data science work (ie. Git, DRY etc.) - Driven & excited by forecasting problems; you care more about making great forecasts of the future as pragmatically as possible than you do about any particular statistical methodology or library.

Keyword skills: Data Science, R, Python

I lead the team - feel free to reach out directly at ai [] coatue .com

This is very cool. I would love to use this in a data science / data engineering context.

At my firm, we build complex data pipelines (data goes from S3 -> Spark -> Redshift -> between Redshift tables etc.). Our data sources comes from FTPs, vendor databases, vendor flatfiles, etc. Further, business analysts build Tableau dashboards on materialized views or temp tables. Data scientists use raw data tables and create aggregated tables or forecasts.

It would be tremendously helpful to have a way to map out programmatically where the data is going and coming from (which vendor FTP did this table come from, where is this view coming from, etc.)

We have utility functions/libraries (use Scala & R) that we always use for reading data, writing tables, etc. It would be amazing to just plug in API calls into those utility functions to auto-generate maps which we could later edit/document. The new trends of data catalogues (see https://alation.com/ or http://data.world)is really useful - but it requires a lot of manual documentation (non-programmatic) that data scientists inevitably don't do.

I think this is exactly correct. Frequency of updates (and clear documentation of the lag relationship between when data is reported and for what period data is applicable too) is often missing or hard to find.

The value of increasing the cadence of updates should also not be understated! A lot of public dataset report on annual frequencies with more than a quarter of delay... Although this is a different issue altogether that has more to do with the processes of the reporting agency.

Coatue Management | New York, NY / Menlo Park, CA | Infrastructure Engineer / Front-End Engineer | Fulltime

Coatue Management is a hedge fund based in New York City.

We're in the middle of building a market intelligence platform based on non-traditional data (ie. non-financial data). We look for creative sources of data (everything from e-commerce pricing to satellite data - typically realtime and granular), use it to model key firm and economic metrics. If you're interested in large scale data pipelines or Bayesian statistics, we have interesting challenges.

Data Engineer:

    - Experience in using and tuning Spark on large datasets (>10TB) for batch processing

    - Preferably has experience with AWS EMR and/or has used Scala for data engineering

    - Keywords: Scala, Spark, Redshift, Docker

Algorithmist:

    - Experience in Scala (or any other functional language e.g. Haskell, F#, OCaml)

    - Strong algorithms and data structure background e.g. competitive programming

Front End Engineer:

    - We're currently looking for a front-end engineer to build out interactive dashboards. You would work with both the data scientists who generate output, and the investment staff that consumes it, to build the most effective visualizations possible.

    - Example dashboards include tracking the historical accuracy of various datasets compared to the firm metrics they measure, drilling down into the constituents of a macroecnomic measurement, and visualizing economic data with spatial and temporal dimensions

    - This role offers the opportunity to take ownership of the architecture and tech stack, and it offers freedom and creativity

    - Keyword skills: Bootstrap, D3.js, Chart.js, Plotly.js

Data Scientist:

    - Experienced at SQL or Spark, strong at a scripting language e.g. R, Python

    - Keen to not only do ad hoc data diving but also build automated tools for repeating that analysis.

    - Employs best practices of software engineering in data science work (ie. Git, DRY etc.)

Email us if you're interested or want to learn more: hr [at] coatue.com

You might want to aim at industries that are seen as less desirable or trendy but have real technical problems that you'd be good at solving. It's even better if you are directly tied to a revenue generation process. Hedge funds come immediately to mind.

Could you go into detail about how you are technically advanced relative to your competition?

Would be curious to hear what that entails / differentiates you from other investment firms in the same space. The alternative data space taking off among hedge funds is pretty interesting, and I'd be interested to hear how a real estate fund approaches it.

Nontraditional data is THE future of investing and engineers are best suited to find it. For instance, if you wanted to know who was spending the most to acquire users, wouldn't it make sense to get bid data from ad exchanges? If you wanted to know which were the most anticipated games of the year, wouldn't the Reddit API be the tool you'd use to check? And if you wanted to predict crop yields, wouldn't satellite data of crop fields be the most realtime way to see it? Traditional equity research doesn't do this - but we do and we've seen it work. We're building a system to systematize our capabilities.

The scale of what we're trying accomplish is no easy feat - we handle realtime data at very high throughputs and analyze datasets > 1TB in an environment where accuracy and speed counts. Our output has a real impact on investment decisions.

If this sounds interesting, email us!

Is this associated with http://www.222labs.com ?

If not, is there an email or somehow I can PM you or @ajvb at?

Looks good. Very much looking forward to this being production ready. I am currently looking for a scheduler that provides a better UI/UX than cron alone.

Currently, what do you all use for this sort of thing?

Nextdoor has a not-yet-open-sourced solution: https://engblog.nextdoor.com/2015/06/10/we-do-not-run-cron-j... that looks great.

Does anyone have any experience with Yelp's tron? https://github.com/Yelp/Tron

Cool, very interesting but is this data just scraped from AngelList? If so, is there a risk that its more representative of AngelList's growth as a platform / relevancy amongst employers?

This is interesting, it's actually uses by the SEC as well [1]. I think it still does not have 100% adoption though.

[1]. http://www.sec.gov/Archives/edgar/data/1288776/0001288776140...

A related question - how you design a database for something as complex as a tax system? Given that tax laws can (and will change), that changing legislation often adds possible combinations to any field, and that there is surely a every "long tail" of unusual circumstances/combinations.

Is this a good use case for a document store, NoSQL database?

Sure, at the moment, I'm creating a new project and arranging elements on the page to design one group of components (for instance, the main table panel of my monitoring app).

I then start with the "deepest" children that I want control over (beyond them just being Bootstrap components) and begin creating components, each time including children. I generally put everything into a single group. When I am finished, I just take that group's folder and add it to my components/ folder of my project. I then do all the functionality in my text editor (Sublime).

A few caveats to keep in mind:

- I filed issue #24 on Git which you fixed, thank you - very helpful. I have react-ui-builder running on an ec2 instance so I can switch to it between computers and do not have to worry about running the server locally.

- I am a data scientist/statistician (both in my job role and by training), however, I build internal applications for all sorts of visualizations/dashboard monitoring purposes. I do most of my backend programming in Python (Flask) and have been using React for the front.

I hope this helps. On the flip side, I'd love to hear any tips to improve that workflow. Currently, I'm using react-ui-builder as a sort of "component factory" but I'd be open to using it more extensively.

I'm hesitant to suggest component placement features as this might just be coming from my inexperience building interfaces but a few sticking points I've ran into:

- The 'wrap' component button is useful. It would be helpful to have an equivalent 'get wrapped' button (ie. insert child).

- The up/down arrows for moving components work well as does the 'move cursor one parent up'. It would be useful to be able to do this with a component as well. (ie. make it a child of the following component or make it a sibling of its immediate parent).