seanharr11

Notemeal | Senior Software Dev | Remote then Boston | Full-Time | https://notemeal.io/careers

Typescript, Apollo GraphQL, React, Ionic

We’re integrating dietitian-built meal plans with chef-crafted menus to create a new dining experience.

Join our founding team of hackers in building a modern tech stack that (literally) serves 5,000 athletes across the NFL, Team USA, USSOCOM, NBA, NHL and MLB, with plans to (soon) disrupt the consumer dining experience.

Come work at Notemeal. We have an office with a private gym in Boston and can get tested for COVID once per week as a perk. https://notemeal.io jokes aside (kind of), we were remote for the first year, and it wears on you pretty fast to eat, sleep and work in the same room.

I've begun the rationalization process...I should be able to start this sometime in January. I've begun drawing up the plans though...

We did this with the New England Patriots, only we moved to an on-prem PostgreSQL instance. Specifically, we were on Oracle 8, so just finding the compatible drivers was half of the battle. I wrote a Python utility that carries these (nasty) migrations out from end-to-end. It works well, probably doesn't scale well, and needs tests. IT also doesn't solve the problems of Oracle-specific features, we just dropped those, which is probably not suitable for most folks, though many of these can/should be done at the application level.

Anyway, I made a promise to myself that when the project hits 300 stars I'd write tests and refactor for scalability. I need 3 more. Probably will be looking for some help if anyone's interested... https://www.github.com/seanharr11/etlalchemy

This page suggest that I am logged into "VK", which apparently is the Russian-equivalent of Facebook, but I never heard of it before this. Any reason why that is?

I'm trying to give some consult to a company who is looking to store JSON data in MySQL, and they have a Rails app.

Given a Rails web app (i.e. ActiveRecord models), what is the best choice if the JSON column type is essential?

Initial research shows that PostgreSQL stores JSONB more efficiently, and provides faster query performance with indexes, while MySQL exposes and JSON query interface w/o much optimization. Thoughts?

Just a quick interjection: I wrote a library that migrates between any 2 relational databases. In my case, I migrated off of Oracle and onto MySQL, but the tool supports any RDBMS supported by SQLAlchemy.

https://github.com/seanharr11/etlalchemy

The best way to know is to try! It takes 4 lines of python to get you migrated.

PostgreSQL is a bit more feature rich in my findings, but for the majority of web development there really aren't huge differences. In my experience, MySQL performs a bit better for lots of simple READ queries, while PostgreSQL can handle larger, more complex queries and very frequent WRITEs.

Let me know how it goes...please open issues if you find them!

I wrote an open source tool that migrates between any 2 relational databases.

https://github.com/seanharr11/etlalchemy

This includes automatic migration of the schema, constraints, indexes and obviously data. It leverages fast bulk loading tools like mysqlimport, and PostgreSQL's COPY FROM. It doesn't need any config. Just plug and play.

The tool uses SQLAlchemy to abstract column types, and handles various column type conversions between any flavor of SQL.

The one thing it needs are tests, and scalability support for tables bigger than a few GB!

A few of these steps could be solved with this tool, including schema/index migrations, and even initial (fast) data transfer: https://github.com/seanharr11/etlalchemy

Ahh I got it. Understood, definitely worth clarifying, will update. Thanks.

The comment right below has 2 links:

Gradient Descent explored similarly to this post: http://thelaziestprogrammer.com/sharrington/math-of-machine-...

The difference between the two: https://www.quora.com/In-optimization-why-is-Newtons-method-...

Thanks for the feedback, I co-worker just jabbed me with regarding the log property mistake also...

As to the motivation for the correct step: can you point me to a resource that explains this? Not sure I follow...

I just took the index of the data point (i.e. 1, 250) as the y-axis with the intention of "Stretching out" the data set along the y-axis. Otherwise the data would otherwise be illegibly compressed on a 1-D number line.

In retrospect, I could have better represented the data with 2 overlying histograms, but this (somewhat) captures the intent of showing that "more expensive houses tend to have more than 2 bathrooms".

I am a programmer trying to learn math, so are my intended audience members.

That said, I should include facts related to convergence, and maybe even speed compared to SGD.

As to the reciprocal -> inverse generalization, do you have any resources you could point we towards to better understand this?

Additionally, a concrete answer to "Why would following the tangent repeatedly be a good idea?" has been hard to come by for me. I am able to visualize this, but if you have resources that explain this well please share.

Thank you. I err'ed on the side of caution to (try) to reduce information overload, but I also didn't want to assume too much prior knowledge. Delicate balance...

Maybe a linked post that dives deeper into the 5 steps of Newton's Method? Would that be more approachable?

I wrote this post (first) on Gradient Descent that will be helpful if you liked the OP.

http://thelaziestprogrammer.com/sharrington/math-of-machine-...

This looks at the algorithm the same way that the OP does (mathematically, visually, programmatically), only it applies it to a simpler Linear Regression model.

Interesting. That's also been my experience so it's good to hear confirmation. It seems the be consistent with the posts I've read on how "rank" is calculated.

I wonder what the penalty is for "re-posting", especially because timing seems to be key. You post at 8:30AM and get very few looks from your target audience, and then post again at 10AM and coincidentally reach a large amount of your target audience.

I wonder where the line is drawn between "spam" and "persistence"...

For those of you experience a github outage (like myself), you can read about the software package here: http://thelaziestprogrammer.com/sharrington/databases/migrat...

Down in Boston, MA

Harold Martin held without bail (high risk of flight) accused of theft of 20 years worth of government (NSA) tools/data, Trump stating he will not concede the election, tens of millions of IoT devices used in DDOS attack, Assange (wikileaks originator) cut off from internet, DNC hacked and exposed.

A conspiracy theorists dream.

Did you get a chance to try out etlalchemy? Please feel free to share the tool with friends/colleagues that may benefit from it!

My experience with my current employer is that Oracle/SQL Server/Windows are used because of the lack of trust in open source solutions, usually by management who grew up in the Stone Age of Software.

All 3 of the aforementioned slow EVERY process in the building down, especially when it comes to application development.

1. We (an NFL team) had an Oracle 9i database (which Oracle pulled support for in 2010) storing NFL player's Scouting information for > 20 years.

2. We needed to make the switch to a modern solution because as of June 2016, we had been running w/o support for 6 years (I got here in 2016), and the RDBMS (9i) was quite slow and not compatible with modern drivers and applications.

3. Our biggest technical hurdles were unique to the tool that we used for migration (etlalchemy, described in (4) below). i.) Finding the correct cx_Oracle (Python) driver to communicate with the correct Oracle Instant Client version, and then finding the exact version of Python libraries which supported that driver (b/c Oracle pulled support for 9i so long ago, finding the old instantclient binaries was nasty work). ii.) Handling FK constraint violations when importing data into PostgreSQL. Because you can't turn constraint checks like you can on MySQL (SET FOREIGN_KEY_CHECKS=0;), data that violates constraints must be resolved before the migration. As a solution, I added support to etlalchemy (the next bullet below) to "dump bad rows" to STDOUT, and skip the foreign key that is violated during the "constraint migration" step.

4. Regarding tools, we used https://github.com/seanharr11/etlalchemy to facilitate the migration. I created this open source tool 2 years ago, published it to github, and used it to carry out 99% of this project. The only thing that had to be migrated manually were the Oracle functions (most of which we tossed out as they were used for ColdFusion/Report Generation). The most powerful feature of this tool is its ability to migrate schema (including column types) between 2 different RDBMS's. It also loads data quite fast leveraging PostgreSQL's 'COPY FROM' bulk import, and MySQL's 'LOAD DATA INFILE'. (Constraints and Indexes are also migrated with etlalchemy out-of-the-box)

5. From a business standpoint, we are just completing the project now and are likely going to choose MySQL as our final destination. This is mainly because our database is very OLTP-oriented. Our applications perform very small SELECTS, INSERTS and UPDATES to our database at high frequency (think football scouts updating player statuses, live stats feeds from the NFL, etc...), and InnoDB (MySQL storage engine) happens to do a very good job of facilitating this type of database (high-frequency simple updates). PostgreSQL would have been a better choice if we were to run less frequent queries involving more computation (OLAP-oriented), but we don't (yet) have a huge Machine Learning/Data Mining requirement here. From experience, PostgreSQL has a huge toolkit of functions, very cool indexes, and other nice-to-haves, we just don't need them for our business case. Both choices have a much stronger community, and answers are found via StackOverflow/Google rather then reading the Oracle manual until your eyes bleed.

6. If you want to read about my experience, I have posted a quick write-up here, including some more background info on etlalchemy: thelaziestprogrammer.com/sharrington/databases/migrating-between-databases-with-etlalchemy

If you decide to give the tool a try (it is battle-tested), please let me know how it works for you. And please, if you have any questions/feature-requests/issues, reach out to me. I am trying to get some momentum going for this project, as it would solve lots of people's database migrations problems as it solved mine.