un-devmox

That's the debate I was trying to initiate!

For instance, is it OK to crawl a blog with explicit copyright, save that data, then publish it elsewhere?

> If you put content up on a public site, anyone has the right to go and look at it.

Fine.

> Stop complaining when someone saves it.

Fine.

What you don't say is that it is fine to recreate and publish that content against the owner's wishes, especially when said content is copyrighted in one way or another. You're failing to see the whole picture from the content owner's point of view.

Project: an app that reads pdf table data from blood panels to better display trends in values.

Problem: it sucks to have to flip around various labs to compare results.

Also, trying to make a better search engine for fine-art that is for sale.

Take, for example the referee of the KY game who got his business trolled as well as death threats. I don't think he did anything to "feed" the trolls however he may be irreparably damaged by such people. I don't think you can ignore them and hope they just go away when your business is threatened as well as your life.

I haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing.

As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.

Ergonomically, IMO, using a mouse or track pad is much worse than typing. That said, I am a very deliberate and methodical developer.

What always bugs me is that finding complete documentation of keyboard shortcuts for whatever program I'm using can be a battle. Better documentation for many apps would be greatly appreciated.

As a side note, I've battled carpal tunnel and tennis elbo for years. One thing that has helped, besides keyboard shourcuts (and many other things) is using an upright mouse on my right and a track pad on my left.

I have, according to my MRI, severe cervical stenosis. I also have carpal tunnel, which was diagnosed first. Based on my symptoms and the awful looking MRI, 4 highly regarded spine surgeons recommended a 2 level fusion right away. The 5th doctor, instead recommended carpal tunnel and ulnar nerve releases. My symptoms abated by 80+% and is something I can live with comfortably.

None of the 4 other doctors even considered dealing with my carpal tunnel first, which, combined with stenosis causes what's called a 'double crush' scenario. Only one surgeon recommended the least invasive and risky procedure for me! Even though a fusion in the future is likely, for now the risk - reward for such a procedure does not warrant immediate action.

I also think it's worth noting that that 5th doctor is the only female of the group. She was the only one that didn't seem gung-ho about the fusion. I'm not saying gender plays a role but...

To be fair, Canada doesn't have the rosiest history in regards to immigration. My point being, if they can change, so can we.

https://en.wikipedia.org/wiki/Chinese_Immigration_Act,_1923

https://en.wikipedia.org/wiki/Japanese-Canadian_internment

http://www.cic.gc.ca/english/multiculturalism/holocaust/hist...

Thanks for the link.

Interestingly in 2015, AG to be Sessions introduced an amendment that "would have exempted federal agents from the requirement to secure a warrant if the government asserts that an emergency situation exists."

The bill was then withdrawn from consideration.

Is there something similar attached to the current version?

This is another troubling action meant to chip away at what government transparency we have left. I'm sure that the argument could be made that government agencies are "inundated" with requests that they can't keep up with and all this is too "costly."

This is worrisome! Will the feds start charging fees to view electronic docs like the State of Wyoming? http://www.wyomingnews.com/news/wyoming-court-ok-to-charge-f...

Access to information is being attacked at the State level as well.

http://www.wyomingnews.com/news/wyoming-court-ok-to-charge-f...

Wyoming is already the only State to tax wind as far as I know.

edit, adding a reference: http://www.hcn.org/articles/wyoming-considers-raising-an-alr...

Much of the state's budget is derived from the coal and gas industry. Hence, all our other taxes are low. For the past few years we have been in a budget crisis due to the drop in energy prices. The university and public schools in particular are having a real hard time as their budgets continue to be cut.

I'm not sure whether or not if this will pass. Given the current political climate and how broke we are, I've got a bad feeling.

edit: adding source of info about budget shorfall: http://www.wyomingnews.com/news/gov-matt-mead-says-million-i...

For those that don't know who don't know who Vincent Canfield is (I didn't), it is worth reading up on the man as he is a fascinating if not controversial person with a flair for raw/abrasive humor and a confrontational manner. He faced extradition and criminal charges in Germany [0] and his email service, cock.li, was at the center of the terrorist hoax against US schools [1].

[0] https://twitter.com/gexcolo/status/715640881651564544?lang=e... [1] http://motherboard.vice.com/read/the-story-of-cockli-the-sit...

I would like to respond to a few comments:

> Why do we only hear stories about this from the US and not from China or Russia? https://news.ycombinator.com/item?id=13454966

As a proud US citizen, I understand the slight you are feeling. It's as if we are continually being picked on and singled out for every misstep. In these types of thread I always see comments like:

> USA: make East Germany great again. https://news.ycombinator.com/item?id=13454689

> Yip. Traveling to the US is ironically the one place where I'd be concerned about my rights & freedom. Their tourist harassment force really needs to relax. https://news.ycombinator.com/item?id=13455194

Which infuriate me! But, we (as US citizens) should be held to the highest of standards because at times what we preach as a country is contradicted by our actions. It's not good enough to say, "Well, this is an isolated incident," or "Look at what x_country does." To justify abusive over reaching actions by our government based on what another country does is not logical or constructive. What we need to do is improve ourselves as a country and by doing so maybe set an example for others.

> Looking to government for protection from exploitation by employers, polluters, etc. is not what a large percent of Americans do.

I think some of the reasons for this is due to the immigrant history of the U.S. When my grandparents immigrated to America their only desire was to be free from government persecution and harassment. They left Europe with a wary eye for government, and rightly so.

I think many in the U.S share a similar experience whether or not they are 1st or 10th generation Americans. Unfortunately, that attitude has manifested itself in some negative values and policies.

< Some of Toyota's modern car plants employ a meager 1500 people which is about and order of magnitude decrease from what car plants used to employ.

1500 jobs + the peripheral jobs (as a another commentator pointed out) could be a boon for many of the small counties* in which these plants are located.

*https://en.wikipedia.org/wiki/List_of_Toyota_manufacturing_f...

I must have watched the Painted Spider a thousand times when I first started climbing in the early 90s. It was so inspiring! http://www.chockstone.org/media/Review.asp?Action=Show&BookI...

For those interested in the bolt or not to bolt debate, the Dawn Wall has always been at the epicenter of this controversy. When Warren Harding and Dean Caldwell did the first ascent they placed many bolts (most were tiny brittle aluminum dowels). Later, Royal Robbins and Don Lauria, made the 2nd ascent with the intention of removing those bolts. But they stopped removing the bolts as the beauty of the climb became apparent.

IMO, as someone who has climbed in the Valley for more than 25 years and has always been a traditionalist, drilled bolts have there place in climbing. It would be hard to convince me that bolts on El Cap or any where else in the Valley have 'ruined' the massif.

You sound like the Donald talking about Mexicans!

So sorry for your reputation. /s

> ugly {BLABLA} everywhere

It's been a while since I used angular (1.4) but from what I remember, ng-cloak handles this.

Yes, I can see (now) a different interpretation of the parent post. I get your point.

It is too bad you didn't have a good experience with CC. My experience with community college was quite different and very rewarding. In fact, when I transferred to university I was farther along in computer science and math than my classmates. However, I did have one class, WordPress (in which I was told we would learn to build themes from scratch), that parallels your experience!

> He went to school while working. Tons of people do that.

[After losing everything, a man in his mid 40s went back to school for the first time since 1982 while supporting 3 kids and a wife (who also went back to school) working as a janitor and, at the same time, earned an engineering degree from a well respected school.] != [He went to school while working.]

-- So he had significant prior training in a related field.

> He had to drop Calculus 1 initially because his basic Algebra was too rusty. So he self-taught himself through YouTube videos, and went on to ace Calculus 1 -4.

I agree with your sentiment (everyone should be afforded the same opportunity) but, I feel that you are being overly cynical. After all, other employees at the university had/have the same benefits. IMO, this man should be recognized for his accomplishments.

The older you get the harder it is to change careers. Maybe this man's story will help motivate others to follow in his footsteps.

It's interesting that you bring up 'group think.' I immediately thought of my back country avalanche training and how deadly 'group think' can be when I read your comment. I wonder if organizations built around a 'cultural fit' are hurting themselves in the long run.

Thank you!

>The innocent 'I wanted to build a tool to reduce headaches to help the community' is best defense here.

Seems like the best defense as well as the truth. I hope they would see it that way.

>Trying to get approval from a large enough 'chunk' of the data providers without some numbers behind you is a waste of precious time.

That's part of my dilema. It would be hard to get some sort of approval otherwise.

Thanks, I am also considering that type of model as well.

> I don't think the target user will care where the data comes from if it saves/makes them money.

I may also be scraping the targeted user or targeted user boss' website. That's part of the conundrum.

Great advice! Would you build the MVP first before pitching to the data owners? My prototype only indexed (one time scrape) 10 sites and still relies on a fair bit of imagination from the business owner. I'm thinking an MVP would have to index at least 100 or so sites before being at all useful.