885 karma · joined May 23, 2011
I am not seeing Spark being chosen for new data eng roll-outs. It is still very prevalent in existing environments because it still works well. (used at $lastjob myself)
However - I am still seeing a lot of Spark for machine-learning work by data scientists. Distributed ML feels like it is getting split into a different toolkit than distributed DE.
The experience required differs dramatically between [semi]structured transactional data moving into data warehouses versus highly unstructured data that the data engineer has to do a lot of munging on.
If you're working in an environment where the data is mostly structured, you will be primarily working in SQL. A LOT of SQL. You'll also need to know a lot about a particular database stack and how to squeeze it. In this scenario, you're probably going to be thinking a lot about job-scheduling workflows, query optimization, data quality. It is a very operations-heavy workflow. There are a lot of tools available to help make this process easier.
If you're working in a highly unstructured data environment, you're going to be munging a lot of this data yourself. The "operations" focus is still useful, but at the entry level data engineer, you're going to be spending a lot more time thinking about writing parsers and basic jobs. If you're focusing your practice time on writing scripts that move data in Structure A in Place X to Structure B in Place Y, you're setting yourself up for success.
I agree with a few other commentators here that Hadoop/Spark isn't being used a lot in their production environments - but - there are a lot of useful concepts in Hadoop/Spark that are helpful for data engineers to be familiar with. While you might not be using those tools on a day-to-day basis, chances are your hiring manager used them when she was in your position and it will give you an opportunity you know a few tools at a deeper level.
Alternative example: early on in my career, I was a software developer for a PR agency and then a news organization after that. At these organizations, solid writing and editing is drilled just as much as good code review or test coverage is taught in engineering organizations. Their lunch-and-learn sessions aren't about new JavaScript frameworks; they analyze how a particular piece of text was created and find ways to improve it.
My single largest "level-up" in this domain was when I started reading more on character creation in fiction. It forced me to think about how I'm telling stories, which then meant I had to think about how I constructed paragraphs, which then... you get the idea. Some great books I read:
- Robert McKee's "Story," which analyzes how great screenwriting is constructed. You will view action movies in a totally different way after this.
- Corbett's "The Art of Character," which focuses on how characters are created
- Roman/Raphaelson's business writing book, which was recommended by PR legend David Ogilvy
- the U.S. Joint Chiefs of Staff manuals (https://www.jcs.mil/Library/CJCS-Manuals/). These are rigidly structured texts with strange language that forced me to consider writing for a different audience than I normally would (military officers versus tech engineers)
I'll add on that it is tough to find someone who is a great writer and a great engineer early on in their careers. You just haven't had a lot of time to practice both. When you're looking for junior candidates, they will often fall into two camps: they'll write too much, or they'll write too little.
If you're the candidate: err on the side of writing too much.
If you're the manager: writing too much is considerably easier to fix than writing too little.
I think I likely had 25-35 conversations for every bit of paid work, and probably an extra 100-150 of handing out pre-made marketing materials or giving talks before that. The sales funnel takes a lot of time. You don't necessarily have to like the sales process, but you should at least be able to tolerate it or consulting will be very hard.
I would suggest:
* getting some sort of American and European business entities stood up where it is easy for companies to pay you. I believe Stripe Atlas makes this straightforward now for the American side (but I have not tested)
* writing blog posts, recording YouTube videos, giving talks at conferences - anything to get your name out there. You want something where you draw attention to yourself and people can say "who is edem?" and then look up your website and discover you have a consulting business
* figure out how to get a list of 200-300 people with buying power at their organization and reach out offering services. Target your conversations to their needs, like, "hey I see you have job opportunities for Big Data Architects, would you be interested in some consulting work to bootstrap your efforts?"
* be prepared that sometimes you won't get quite the consulting work you want. Sometimes you might hope for a software engineering consulting role, but you might get asked if you can advise on how to hire people like yourself. Money is money.
* ultimately, work hard, try lots of things, be flexible. Most attempts probably won't work, so the ones that do should pay you very well. If you are doing 75% sales and marketing at first, 25% coding... that sounds about right.
If that doesn't interest you, Solutions Architect (or even Sales Engineer) might be a better fit. These are pretty common roles at medium to large companies, but they come with a lot more assistance in the sales and marketing pieces.
#2 suggestion: Chicago, IL. Museums (check), bars (check), restaurants (also a Michelin city), nature (Lake Michigan).
#3 suggestion: Boston
I will 100% grant that the weather in the Bay Area is generally better than all three of those places - but if you don't highly rank outdoor activities, does that matter as much?
We handle payments and customer management for some of the largest subscription brands in ecommerce. If you've subscribed to Native, Hubble, Lola, Soylent, or Billie products, you've bumped into us!
We're a remote-first company distributed across 10 countries (at the time of this post). If you're curious about how we work, check out this blog post from one of our infrastructure engineers: https://eng.rechargepayments.com/a-day-in-the-life-of-a-rech...
Our tech stack: Python/Flask with Vue.js on the front end.
If you're curious, please feel free to shoot me a note: jmosby at rechargeapps dot com. Cheers!
https://www.willmeng.com/goodwill-store-moving-to-shopping-c...
https://www.blueoceanstrategy.com/what-is-blue-ocean-strateg...
If your role is to hang out at your desk until someone sells your implementation services, then no, you will probably not get commission on sales. You should demand a fixed salary comparable to if you were a SWE working on the product directly.
If you are expected to participate in the sales cycle in some way (like stitching together a demo), you will probably receive some sort of commission.
This is a pretty fair thing to ask about in an interview or in comp discussions. "Am I expected to be involved in sales efforts, or am I simply deployed after the sale?" Adjust salary/commission demands accordingly.
Chinese economic restrictions generally tilt toward keeping the majority of capital flowing throughout China and strategic investments abroad. They are definitely not geared to someone who wants to start an English-learning school and repatriate earnings back to the UK or wherever. WeChat/Alipay support this - you can't really spend that money anywhere except in China.
1) Read a lot. Read different styles. Read the newspaper, read fiction, read blog posts, read history textbooks. Read lots of things. The diversity of styles will give you ideas on how different things can be communicated well in their own context.
2) Analyze what you read, just a little. I started with just asking myself "who is the intended audience of this writing?" Asking a consistent question in your analysis will make comparison easier. It will also help you think through your own writing: "am I writing this for the marketing person who is going to try to sell it? am I writing it for the coder who has to install it? maybe I need two separate documents for each of them?" That thinking about writing will help you analyze and adapt your own work.
* It doesn't need much in the way of dependencies to run. If you can get Java onto a machine, you can probably get NiFi to run on that machine. - That is HUGE if you are operating in an environment where getting any new dependencies installed on a machine is an operational nightmare.
* It doesn't require a lot of overhead. Specifically, no database.
* You can write components for it that don't require a whole lot of tweaking for small changes to the incoming data. So, if I have a machine processing a JSON file that looks like XXYX and another machine processing a nearly identical JSON file that looks like XYXX, the tweaks can be made pretty easily.
So, if you're looking for a lightweight, low overhead, easily configurable tool that may be running in an environment where you've got to run lots of little instances that are mostly similar but not quite, NiFi is great.
If you are running a centralized data pipeline where you have a dedicated team of data engineers to keep the data flowing, there are better options out there.
We tend to inflate problems that touch us directly and deflate problems that we don't see very often. Consider if I am the government official responsible for making sure roads are paved. The roads that I use every day (to commute, to go to the grocery, etc.) are likely to be pristine. I will make sure of it. The roads I rarely see will be dealt with on their regular schedule, and maybe not at all if time runs out.
However, if paving decisions were made by a large group of citizens, each of whom had the ability to just go out and pave the road themselves if they felt like it, the city would be much more paved.
The same allegory works for closed versus open source. Consider if LinkedIn had left Kafka as closed source. If we looked at the code base, we would likely still find a pretty good message bus, but it would be optimized for the nuances of LinkedIn's system architecture and proclivities in passing messages. Once it became open source, developers from Facebook and Google and Startup #1 and Manufacturing Firm #2 and so on all jumped in to solve problems. They achieved consensus on how those problems should be solved and pushed forward.
Two points on this allegory that are relevant to Andreessen's argument: everyone in the infrastructure committee had skin in the game (they wanted problems solved) and they had the capacity to go deal with the problems themselves. Andreessen argues we have a lot of the first already but we don't have nearly as much of the second as we need.
Most of the intro decisions around moving to the cloud happen before Amazon is ever involved in the sales cycle. There are some managerial accountants and IT people getting involved to evaluate options in a process called "Evaluation of Alternatives." At some point, Amazon probably started hearing that the IT people were saying "we can't possibly move all of that data to the cloud, it would swamp our network or take five years just to finish the network transfer."
Enter Snowmobile, which cuts that conversation thread off at the knees. Amazon can go to a Fortune 100 company that is evaluating a move and honestly claim that they are prepared to move exabytes of data, physically, into the cloud. And it's not some premium feature that they're going to get charged bajillions for, it's a stock feature of AWS. (Even if it costs a lot of money)
Snowmobile isn't targeted at the average customer, it's targeted specifically at the accountants, IT architects, and management consultants evaluating the decision to move all of Fortune 100 Company X from on-prem data centers into 100% AWS. One extra hug from Amazon to let the most risk-averse person know that they have it covered, and would they please go ahead and write their 9-figure check to AWS.
e.g., AT&T could provide perfect service to the home endpoint, but the customer bought some aftermarket router from their cousin, who had configured it for Verizon. Customer calls AT&T to holler. Tier 1 support doesn't know what that particular router config GUI even looks like, so it gets bumped to T2 or T3. Ultimately to find out that the customer's cousin had hardcoded DNS to some internal Verizon system that's not visible to AT&T.
Repeat x100K. ISPs job isn't just "provide the Internet," it's also "provide all the troubleshooting for every non-technical customer who just wants to watch Netflix but doesn't even know what a router is"
Anecdotal disclaimer: I have been both an OSINT analyst and a developer supporting OSINT work in the past. I understand completely why OSINT analysts like to do things by hand - they can understand where the data comes from; they can cite sources as they go; they can tailor things specifically to their use case. Understanding the process is indeed important to knowing how we got to the conclusion.
And yet, we're frustrated when there's a panic situation and we need a report of any kind in two hours. We're frustrated when we're handed sixteen targets all at once and have to suss them out over the weekend. We're frustrated that visualizations suck and we have to custom make them ourselves. Better tooling, an opinionated process, and automation helps solve all of that, or at least advance things a bit.
Can't have anything perfect (entity resolution as #1 on the un-perfect list) but getting a tool together as well thought-out as this one is an excellent step forward. Kudos, @smicallef.
Threat analysis does not play well with common types of AI/ML applications because the problem space changes so frequently with the design of avoiding predictability. You can gain "awareness" of events using AI/ML (e.g., this road suddenly has tanks moving across it), but it's tough to call that a "threat" (maybe other intel told the analyst those tanks were supposed to be there). Compound that problem to "the world" or even "an entire country" and you run out of compute real quick.
Most solutions err on the side of just popping up alerts but don't identify "threats" until an analyst looks at them.
The "Please stop. I've got this." is an action-oriented message that is conveyed before the action. It says "I would like for you to do X because of Y." It is conveyed in an imperative structure, which is used for a command or a request. "Please stop." <- request. "I've got this." <- followed by a rationale for your request. The request is conveyed first, followed by the rationale for why you are making it. "Do this. Here is why." The "I've got this" also carries succinct subtext of "I agree with you that our son crying is a problem that someone should take care of, I just think it should be me instead of you."
Conversely, when you say "I was going to get it," it is a declarative message rather than an imperative message. It does not tell your wife what you would like for her to do. It is merely additional data that does not help her make better decisions, which means it is mostly useless for her. An extreme comparative example is looking at a person who has just burned their hand at the stove and then saying "that stove appears hot."
Humans don't like to receive an array of "reasons" data before they've received the "request." We don't like statements in the form of "because of Y, Z, A, B, C... please do X." Our subconscious can typically detect that we are receiving a stream of "reasons" and we are starting to brace for the "request" (because the request might be unpleasant).
People like to know what others expect of them. Your comment says you would like to know what your wife expects of you. When you follow the <command> <rationale for command> order of operations you are giving people what they want (what you expect of them) followed by your reason for wanting that. When you go in the opposite order, you are conveying the converse: "I want something of you but I haven't told you what it is yet."
As an anecdotal example, Uber Eats/Grubhub/Seamless fees are a material portion of total ticket price in DC (which seems to be different in other cities). They are also rarely subsidized for customers here. I am still eating at the same restaurants and probably get take-out just as much as a New Yorker where delivery is way more common - but I just walk over to pick it up.
Similarly, when Uber is surging, I take the bus or the metro.
Part of what makes cities so appealing is the diversity of options. If consumer-tech-based apps get more expensive, there are actually alternatives to literally all of them. I don't need a Peloton - I can just get a gym membership somewhere. Or buy a bike.
100% agree that Millennials are likely about to change their lifestyle options. But not certain that "change" directly equates to "accept higher prices."
The initial round of California VC money came from hardware startups cashing out. Hardware is tied to the physical realm, and thus locality is important - if you need to buy leftover equipment or supplies, it's way easier to get them if you're in the neighborhood as your vendor.
The next round of software startups in the 90s through early 00's had a similar switching cost with VC funding. Though we obviously had telephones and email, we had not fully transitioned into the "remote work" mindset that allows teams to collaborate regardless of distance. It was challenging for VCs to work effectively with their founders, and for founders to work effectively with their teams - which forced teams to co-locate with founders, and founders to co-locate with VCs.
You don't need the world's best software engineers to launch a startup - you can launch something with sufficient complexity to start making money in Denver, or Austin, or Chicago, or anywhere else with a reasonable supply of engineers. And now, VCs are far more tolerant of remote founders given how easy it is to stay in communication. If all that is true, why make things more economically challenging than they need to be by trying to fight the economic headwinds of California?
"I’ve lived in the Midwest, on the Texas/Mexico border, and in New York, and when I moved to San Francisco..."
Additionally, the argument in the entire piece is about California not being the epicenter of all of the things it claims to be.
If this type of job is your jam, it has the endorsement of this super-impressed and super-happy customer.
I also received some pretty sound advice around three years into my career: "just assume that people mean nothing more or less than the literal words they said to you. Don't read more into it than they actually say." I found that if I felt awkward about a situation, I was trying to read in between lines to find some reason that a person secretly hated me or were annoyed by me. They had never actually said or done anything to indicate that they even thought twice about me once I walked away, but I made up all sorts of stories about them in my head.
The competitive set is differentiated in some of their niche work (like specific-device-embedded maps, such as the ones in Tesla), but the first principles for the common developer are all the same.
Crystal City already has a strong tech presence - it's government, not industry, but still - no shortage of technology around. 25K new workers is about a second Pentagon in the area, but nothing more than that. Office rental rate pressure will slightly go up, but Amazon's a drop in the bucket compared to federal government buying power for office space.
Queens, though, is a dramatic change to the general landscape of that area. I can't even imagine an Amazon-style campus in the middle of Jamaica.