Tackling Webdev as a Bioinformatician: why is it so hard?
jessimekirk.com
jessimekirk.com
"I am a professional in Field X but I am doing work in Field Y despite having minimal skills or experience and I ran into loads of problems and issues that I don't know how to fix! Field Y is total bullshit and people need to make it easier so people with no skills or experience can do it! Wah wah!"
... or to be less facetious, modern web/frontend development is a genuine skill set that benefits from experience and knowledge.
Should we expect anyone to just jump in cold with minimal experience and not find it difficult? E.g. in the same way would we expect people to start bioinformatics cold and be productive immediately?
I'd doubt it.
On the past web dev has been looked down upon as somehow "lesser" than "real" engineering, with sneering criticism from people more used to traditional development approaches (particularly criticism of JavaScript). It is just some web pages right? HTML and CSS? How hard can it be?
Well guess what: it is a genuine skillset and people should not expect to just be able to jump into something without knowledge training or experience and be proficient at it, just like we'd not for brain surgery, car mechanics, concert piano, structural engineering, or indeed bioinformatics.
</rant>
I have some experience in both web development and scientific computing in various fields (not bioinformatics). Things felt just as hard at the beginning in all cases—I don't recall things being any better in scientific computing compared to web development.
I agree that documentation is terrible most of the time, but that's not something limited to web development in my own experience. (On the contrary, check out the documentation of the website in question in this thread: https://news.ycombinator.com/item?id=23031408, it's absolutely fantastic).
Some things become second nature when you have developed expertise in that field, and it's easy to criticise something that you have comparatively less knowledge in and pass it off as "too difficult", particularly when the web development part seems to be a means to an end (which seems to be the case here).
Edit: missing conjunction, tidied up the last sentence for clarity.
If anything, programming for academia is more of a mess than web development. Case in point - I spent the better part of a day rewriting what's essentially a parallelized for loop because the library I'm using in Julia to open HDF5 files has a well known memory leak. I have the impression that something similar in any popular web dev framework or library would have been ruthlessly purged a long time ago.
You can get the data out of an HDF5 file with
using HDF5
c = h5open(filename, "r") do file
read(file, "data")
end
Bug report: https://github.com/JuliaIO/HDF5.jl/issues/349
It's annoying it isn't fixed and working around it is undoubtedly annoying. This seems different from a system where there's lots of complexity in just choosing what frameworks to start with and if/how they play well together, and all of that rapidly changing over time. (I'd bet, for example, that many of those examples used to work).
Sure, but this also applies to understanding the U.S. tax code, which I think is a needlessly overcomplicated, poorly designed system that requires layers of inefficiencies to extract value from, much like web development.
The complexity in webdev exists for good reasons much (most?) of the time.
If you want interactivity, you need an actual programming language. If you're using an actual programming language, then there is a lot of tooling that can be used to make your life easier in the long run. Not intially, but definitely from a maintenance / bug-fixing / etc perspective. If you want to work on a large, complex project with a large team, this "complexity" is invaluable.
That being said, JS as a language kinda sucks still, but "complexity" has been added to attempt to ameliorate that too, such as with TypeScript / CoffeeScript / etc.
(That is, the web is a distributed system and distributed programming is hard, sure. But the configuration hell you often run into is separate from that.)
This is probably why the author is struggling. This is a real world industry tackling real world problems. It is not the neat and cut world of science research.
You had me up until this last sentence. Everything I've experienced in science research is the opposite of neat or clean cut. Actually, I think I do my best programming and debugging with a treat it like a science experiment.
Its a testament to the awesomeness of the web however that so so many people even without any formal training can go about creating value.
You _can_ be a webdev tradesman, fixing things till it works, kinda, or you can delve into the fundamentals and be much more productive. But you don’t have to and I think thats awesome.
For example I still remember that conference that introduced Ruby on Rails to the world so many years ago. Most webdevs did not know there was an underlying theory and standard (REST) that can be used to make web apis bearable! After rails, we kinda figured huh, so this RFC for the web was sitting there all those years gathering dust and only weirdos and graybeards talked about it. But it was actually very very useful.
That last 1% actually have read the OG paper and when they say REST they mean REST and its real theory.
"I am a biochem major but I decided to work in bioinformatics/scientific-programming despite having minimal skills or experience and I ran into loads of problems and issues that I was able to steadily solve because the ecosystem felt sane, the tracebacks were clear, and the debugging usually had a clear path forward. Then I was a bioinformatics professional doing work in webdev [...] I ran into loads of problems [...] that felt consistently more opaque and magical than when I was learning scientific programming"
I think that's the heart of it. The transition to the web feels so much more chaotic than the transition to bioinformatics.
I fully agree with the article's opinion that things have become needlessly complicated.
The new best practice seems to be to install NodeJS, then use the npm repository manager to install bower, then use that to download your packages. And 5000 dependencies and 500 MB in downloads later, you have bootstrapped a Hello World template.
Or name me one person who can, from memory, correctly configure the rails sprockets asset pipeline :p
There's no denying that most of the tutorials and contemporary web development advice boils down to "follow these instructions carefully + magic happens"
Also webpacker for rails is somehow both insanely complex and alright to use.
To this I'll only add that "follow these instructions carefully + magic happens" works only for ~6 months after a tutorial is published, longer than that and some dependency or toolchain update somewhere is bound to break the instructions in a way incomprehensible to a novice.
> Or name me one person who can, from memory, correctly configure the rails sprockets asset pipeline
I feel that one, it's insane that the most basic "I need to execute a few lines of very specific JS on this specific page" must devolve into a DRY absolutist abstraction madness, previously with sprockets, now with webpacker.
I remember when Rails first hit the ground. It was touted as a revolution in Web simplicity (and was relative to then, when what we expected of our apps was relatively simple).
But, someone got it in their heads that web apps should be near-indistinguishable from desktop apps. But, let's not address the impedance mismatch there. Instead, let's create a bunch of layers out of new tools and concepts to obscure the fact that these things don't fit together.
So our projects/apps now demand more and have just grown extremely complex. Turns out that making the Web act just like a native app (while battling all its Web semantics) is hard. We want it to be X, when all it wants to be is the Web.
Add to this all of the other disparate pieces that we want to work together: countless technologies, languages, libraries, 3rd party services, etc. And if you're talking full stack (and who's really not fullstack these days?), the number of universes that you must make coexist is insane. And, much of this is to layer over existing technologies (like Rails) that weren't intended to be used in this way. Enter entire concepts like transpiling, webpacking, etc.
Then to deploy, You're probably dealing with no less than 6-10 cloud provider services, each with its own set of concepts and requirements to be satisfied. Even if you're not the one deploying, you have to understand their implications on your code.
Web development is now less about writing logic and much more about wiring a bunch of existing stuff together, with discrete chunks of logic serving as the business/domain layer glue. I mean the app still has to serve some business purpose, so the code that supports that must be written. But that's barely the start these days.
So if we ask web developers how to solve the above problem we get at least 25 different ways to sovle with with JS , and some other 25 ways if we use some backend PHP,Python, Java. If you ask some Qt/Flex developers you will get the same response, Drag and drop the DataGrid and set the data source to your data (and you get a ton of fancy features like adjustable columns, sorting,,,) and you don't have to install 300 dependencies.
Comparing the entire gamut of web development against a single library seems unfair.
What about asking the same question of .net developers, VB classic developers, golang developers, electron developers, python developers, Java developers, c++ developers, swift developers? They'd all do it differently than QT developers would because there are different GUI frameworks that need to be used.
This problem is not unique to web dev.
so in summary in any GUI framerwork the reaponse is the same drag the DataGrid and connect it to the data(the names can differ), maybe some Delphi,GTK,JavaFX,WPF devs can confirm or disprove me.
I do not think this is the case - e.g. just look at the two different options just for Java (Swing vs JavaFx):
https://docs.oracle.com/javase/tutorial/uiswing/components/t...
https://docs.oracle.com/javafx/2/ui_controls/table-view.htm
This is not just dragging and dropping some common component - each thing is implemented differently, and so has a different API to control it (and require huge tutorials to work out how to use them) so everyone has to do it differently. They're not just different APIs to the same underlying OS-provided control. A Java Swing developer will not just be able to do the same thing they always do just in Qt, TK, Wx, WPF, WinForms, Gtk etc etc etc - it is still totally different. Sure the .net languages are all the same (i.e. same API ... but then do you use WinForms? WPF? Gtk#? others?), but there is more to GUI development than .net.
Replace angularN from above with React or even better Vanila? Make an Ask HN if you want on how to solve the problem with Vanila + vanila library (optional) and make a new question on how to solve it with Qt or WPF (optional with libraries) , as optional ask for sane funtionality like resizing columns, sorting data, re-arranging columns and performance when displaying more then 100-1000 rows.
Another point I really like is that I can view a webpage from the 90’s and it will look fine (maybe not great)
And there are frameworks, platforms and plugins which allow you to do what you described without little coding. You can go up and down on the stack, switch stacks and still get to the same end result.
Most of the time your solution is broken. Some examples I hit recently from giant companies, the PayPal amount input keyboard handeling is broken (Del button was not working), Youtube search inpout with it's dropdown can get stuck open and you are forced to reload the page , when you re-invent the wheel you probably are skiping or gorgetting to implement everything so your custom popup might appear outside the screen, or it does not implement Tab ordering correctly or is not accessible, or is not navigable with arrow keys etc. Where in other GUI frameworks you use the built-in one and on top of that all those GUI framework will let you provide/override a custom Paint function if you want (on top of existing customization) so you are not limited either in any way.
Name three.
I'm not joking. I'd love to know about a data table library that meets following requirements:
- Doesn't commit me, the user, to one of the infinite combinations of ridiculous JS build chains. Instead, it should be accessible as a single-file JS dependency in transpiled form. It may use a ridiculous JS build chain for building itself, but I don't want to care about that.
- Doesn't commit me to one of the few big JS SPA frameworks (which each comes with its own combination of ridiculous JS build chains). I want a table, not the kitchen sink, and definitely not the whole meat processing plant.
- Doesn't commit me to particular styling.
- Can handle thousands of rows on a single page.
- Has the most basic ability to: sort each column, and sort by multiple columns.
- Has the ability to do a per-column search across all data (that all matters if it paginates the data)
- Doesn't interfere with browser search, and/or allows a full-table search on all data if paginated.
- Has the ability to filter out ranges of values. Bonus points for extra type-specific affordances.
- Bonus points for the ability to dump its contents into CSV (file or clipboard). Though this could presumably be handled with a small amount of code on my side.
- Extra bonus points: has some minimal pivoting ability.
I haven't found any meeting these requirements, where IMO these should be the default (and if browsers are ever going to standardize on some data table implementation, I'm going to petition for all of these to be included).
The ideal is that Web Technology would have built-in or if not community made building blocks. If made by community it must have no licensing issue, no dependency hell, no 100 alternatives for each widget. In my experience jQuery UI is the one that got close to this. You had only 1 dependency on jQuery, the community was not fragmented to have 100 alternatives for each thing and the included widgets are not incomplete.
I don’t know in this exact use case, but there are probably building blocks (modules) for his to get very close.
I’ll absolutely agree Jquery is an excellent library that I use to this day. It’s extremely elegant and indeed, it is a good example how it is suppose to be done.
But I really meant a self-contained, fully front-end piece of JS. One you could feed a data table and let the user play with it.
A side issue I'm hinting at is that I believe the features I mentioned should be present by default in any kind of tabular display. Without them, the users are getting shafted - they're served data in front of them (be it item prices, virus stats, or a restaurant menu), but they have no way to explore it on their own. This is but a small piece in a great set of ways in which our industry ensures the users can't ever grow and become more proficient, and then complains that users are dumb. Can't have a bicycle for the mind if pedals and handlebar are not provided.
HTML, at heart, is still a markup language designed to separate a stream of text into semantics and display them. CSS ... I am not even going to go into my feelings about CSS except to say that it suggests that people who look at design principles like C.R.A.P. were not near the discussion and were not in the room for some time. A programming language named after Java, in which it no way resembles, that had to be pushed out in almost no time at all given the irrational deadlines, dominates the landscape ... and we must transpile to it. And we're pushing video over HTTP.
All of the first-mover advantages have locked us into this path and many very intelligent people are busy using their prowess to perform work under some very arbitrary conditions, like not being able to use the letter "e" in writing. While it is a challenge and stretches the abilities, no respite is offered, and so there's a tremendous amount of waste occurring as these initial conditions, propagated down through time, must be constantly worked around.
If you don't think package management or transpilers makes your particular project simpler, then don't use them. They are tools.
Perhaps there is a problem in the educational resources available if people think they need npm and typescript to build a simple website.
> Even for seasoned programmers starting web dev is harder than it needs to be.
Maybe, but is it harder than it "can be", or is it significantly different to other fields in this respect. I don't believe so.
If you think the extra layers of unnecessary complexity noone needs is either (a) unique to web dev or (b) easily removable, then I think you may have a slightly naïve perspective on humans & society in general. A perfectly organised and navigable field is pretty utopic.
Complexity is emergent. Look below any complex system and you'll find a simpler one, but that doesn't mean the higher-level complexity is unnecessary. All the things you complained about have good reasons to exist, invented by people who actually work as professional web developers and need to solve problems.
The real problem is "seasoned programmers" who don't work in web development, but who expect to be able to jump in an master it in a day, based on the fact that they put together a GeoCities homepage twenty years ago, and are enraged and disgusted to find that things aren't so simple anymore. It's absurd. Developers wouldn't expect to be able to jump into kernel development, or database development, and find it to be a breeze.
Yes, that’s correct. In my experience, in those fields the complexity of the solution scales with the complexity of the problem. However, modern web development usually tries to squash every problem under a huge stack of complex frameworks and libraries. How many “full stack” kernel or database developers are there? Complexity is not always emergent. In the case of web development it’s in large parts self inflicted. My point was that modern web development is more complex than it needs to be and you didn’t rebut it.
I can't think of anywhere else this is the case. If you want to access an SQL database, that's easier than it used to be, and better abstractions on top are available if you want them. If you want to write a web API, that's easier than it used to be, and again nice abstractions are available. If you want to process tens of terabytes of data, that's so much easier it's not even funny. But if you want to write a webapp... unless you totally ignore the modern stuff it's vastly, vastly more complex than it used to be.
I suppose this may be my grumpy old man side coming out, but I really don't find modern single page apps to be a better experience than mid-noughties webapps in the vast majority of cases. They're usually worse; in particular they're usually slower, and mess with UI expectations more. And yet they are _vastly_ more difficult to produce. The whole thing seems like it took a wrong turn somewhere.
Anyone who has done significant work on a full scale SPA using any framework (or no framework) would have to weigh whether it was worth it. And that's on two levels: 1) Was the additional effort and complexity worth it for the perceived improvements in user experience that were sought? And 2) were there actually any such improvements or, even, is the user experience actually worse as a result?
Because I have to say that I'm not convinced. There does seem to be a period when AJAX first became a thing wherein the added responsiveness gained from judicious use of dynamic operations with partial page updates improved the experience over full page reloads exclusively.
Then, someone said, "hey, if that's cool, then why don't we make the whole thing dynamic, fighting Web semantics (and standard browser behavior like back buttons and history) every step of the way?
Not so sure that's a good premise or has led to a good result.
Problem is some of those seasoned developers didn't just "put together a GeoCities homepage twenty years ago" (which is itself a weird knock on some devs in the midst of a rant that appears to be knocking people for knocking devs). They worked on "actual" hard stuff that even you might consider hard--let's say kernel or database development or even challenging Web development problems that actually existed prior to the current state of things (audience gasps).
So the question is whether there is a mismatch between the objective and the level of difficulty/complexity? Some stuff is just hard by consensus (and the relative number of people who can actually do it). Other stuff is needlessly complex. Another category is both hard and needlessly complex.
So, instead of parsing the complaints here into a personal attack on certain developer skillsets, then rebutting with the same, it might be more useful to earnestly consider whether things have been made overly complex and can be improved.
Hard concepts are ok for anybody with adult desire to learn. It's the ad-hoc transient social choices (package managers, conventions, ecosystems) that are overwhelming. It's draining.
And yet, even within the web development community, we hear a lot of complaints about how this or that isn't beginner-friendly; or how such-and-such is gatekeeping, and so on. This has been baffling to me. Is it the same in other disciplines? Are engineering, bioinformatics or medicine, for example, expected to be beginner-friendly?
- Finding out the right curriculum by yourself is hard
- Once you have the right curriculum, then you can learn it yourself provided one has the right background. In fact, they even can teach that curriculum to people that don't know how to program or do web dev yet
When I was done teaching NodeJS/ReactJS, I went on to freelance for a company that had a microservice backend in Java. Since my CS degree was a Java school, I picked up Spring Boot quite quickly. I needed to build on top of code and not create it from scratch. That really helped, starting from scratch would've been more difficult and does require a lot more experience (so altering an open-source project might be a good way to practice). After 3 months the job was done and I left.
Now, I know that I don't have a lot of experience, but I honestly haven't seen any hard challenges yet when it comes to web dev at the 3 year old startup level. There were no scalability or security challenges to take care of. The web dev rabbit hole can go deep (and that is hard), but in practice it mostly doesn't (unless you work at Google scale, I presume). Deployment with IBM and Kubernetes was just: read the docs and know how to debug and troubleshoot.
It's not easy per se, but if you start with the right background (CS and a lot of programming courses in such a CS curriculum) and get a good coding school curriculum after that, then I would argue that web dev isn't that hard. There are enough startups where you'd be productive.
Though, to be fair: if I couldn't use a debugger, I'd be totally lost. Using a debugger makes it not that hard. This is also why I had a harder time with learning Webpack.
What I found way harder is wrapping your mind around concepts that seem completely alien. I have this still a bit with functional programming (not covered in my CS degree) or back in the day with databases or certain forms of math.
Reading API documentation isn't part of it which is what I feel web dev is mostly nowadays. Even when you learn another programming language, it's all procedural/object oriented, so you just need to learn the quirks of that particular language. When I first started out I also had to deal with emotional fear of being inadequate. But after your fifth language of feeling that way, you know it's part of the process and you shouldn't take it to mean that you are inadequate.
Sure, learning an entire field from first principals is going to be hard, and to do science you need to understand those principals. But the general concept of "make a dynamic web page" is not computer science! What most people (and the author of the blog post) need is a clean abstraction that allows them to glue a few APIs together with some custom forms.
I'm excited about what Dark (https://darklang.com/) is doing with this idea. As you said, "a clean abstraction that allows them to glue a few APIs together with some custom forms."
I mean, there's a strong argument that it has become too complex for no particularly good reason. It is _far_ more complex today than a decade or so ago, and the results in many cases are arguably worse for the user.
If you compare the _average_ modern react-y website to a Ruby on Rails one with minimal or no Javascript from 2007 or so, the react-y one is far more difficult (and slower) to work on, but is it really delivering extra value to the end user? There are exceptions, of course, but I think for most CRUD web apps things have gone in a very bad direction.
My take on this is that the author had some (relevant) programming experience and thought it would transfer better, hence the frustration.
That was my experience, as someone in the same boat (computational scientist trying to put together a webapp). I have a CS degree and my job involves a lot of diverse coding, from low-level stuff to interface with hardware to graphics/ML/data stuff, not to mention gluing together other people's code.
I nevertheless found it very frustrating to get started with web dev. There seem to be a million slightly different ways to do things and people have strong opinions about which ones are best, so it's hard to settle on authoritative source, especially one that's up to date. Many tutorials bounce between basic web stuff (which I wanted) and basic programming (which makes me zone out). Plus, there's a lurking fear that I'm gonna screw something up in a way that exposes a bunch of data or costs me a fortune in hosting (etc) while trying to make something that I sketched out on three pieces of paper.
This seems a little different from bioinformatics, where a lot of the complexity is baked into the subject matter. For background, textbooks certainly vary in how they present information, but they all pretty much agree: Molecular Biology of the Cell isn't going to tell you that DNA is where it's at, with RNA being a fad vs. The Cell claiming that DNA is outmoded and it's all RNA, baby!
A bioinformatician is someone who has quite serious programming skills on top of their biology knowledge. They are frequently bottlenecked by compute and so tend to have a good understanding of data structures and algorithms. They usually know 3 or more languages and are no strangers to databases, version control, containers, et cetera.
And yet I recognize the feeling from the article: they get stuck in the heap of complexity we (as an industry) have created around a bog standard web frontend. The majority of us are not doing rocket science - the applications I’m working on today do similar things to the ones I worked on five years ago. Yet somehow, the ‘how’ of it, the whole Rube Goldberg machine, tends to get more convoluted every passing year.
I often get the feeling a nonzero percentage of it is introduced purely so web developers can feel they’re “real engineers” whatever that may be. There’s nothing to prove, now stop piling on junk and reinventing wheels.
Edit: from the comments it's clear that OP didn't actually hardcode the connection string.
Also, I would look to configure all components to fail loudly, so that if something wouldn't connect to Redid, my server just wouldn't launch.
I'm not trying to say that I'm better or smarter than OP: I've actually tried learning bioinformatics, but dropped out of uni 14 years ago, realising that my brain is just not created for anything biology-related. Rather my point is, many of these problems (and problems I've seen junior developers struggling with) are not some arcane knowledge that you should just learn by heart (like all of those awful genus names in Latin, or all the enzymes in the dreadful ATP scheme that I still have nightmares of sometimes), but a logical extension of more basic and universal principles that actually make total sense once you think about them.
> Also, I would look to configure all components to fail loudly, so that if something wouldn't connect to Redid, my server just wouldn't launch.
In this case though the OP did not have control over the redis connection handling it was through a dependency that silently timed out.
1. A few years ago, I probably would have made the mistake of actually hard-coding my connection strings, usernames and passwords. I'm pretty sure I did that on my first ever webapp, cause I didn't know any better. So, not defending my intelligence, or taking any offense at your statement. 2. But... That's not what's happened here. I'm setting the redis url inside Heroku's "Config Vars" section, and then reading it in with Django's `env()` functionality, as is appropriate. But the url has seemingly been changed from underneath me, for a reason I still don't fully understand.
Side-note: I used to be under the impression that CS was "harder" than biology, and that anyone who understood CS could pick up biology at will, if they decided. Turns out that's not the case, haha.
Is the Redis add-on attached to the Heroku instance? This is how Heroku knows which instances' environment variables need updating when rotating credentials.
Not my domain of expertise, but cheating and lags in online real-time games (like FPS games) can ruin a game quickly. Also, in-game shops, currency, micro-transactions probably are targets for crackers and exploiters.
I don't know how to explain it, exactly, but worrying about this definitely made development more difficult, even when initially setting up and testing out.
Setting up https, or online payments, etc. is actually quite simple. People with engineering or scientific background have most likely faced much more complex tasks. Security and safety aspects can be more stringent in many other industries, including in software development.
The difficulty of the web, I think, is the proliferation of standards (or not so standards) and technologies: networking, https and friends, APIs (payments, etc), html, css, javascript, php, ruby, java, aws, react, vue, redis, SQL, etc. And they are all evolving independently all the time. This is a right mess and can be overwhelming.
To make a safe website, stop wanting to do everything yourself. Use a webhost that doesn't give you SSH access, or a service like Netlify.
You can see that even in simple cases. Writing a basic single-propose script is trivial; it may end up being just a line or two. As soon as you decide to make it general and reusable, though, complexity explodes: positional arguments are confusing, better name them. Users are gonna expect a familiar format. Need a library for that. Reasonable and understandable names are tricky! What arguments are required versus optional? What if _this_ option is specified without _that_ one? Gotta add proper errors for every bad state! What if the user starts using globbing? Or wants to specify multiple inputs and one output, or vice versa?
Your three-line script is now 100+ lines. And you're not even worried about hostile or abusive users yet!
Writing code for the web is all that--and much more besides.
Maybe Webdev is just complex? But tbh, as a bioinformatician (in Next Generation Sequencing mainly) I actually find Webdev easier. I mean, we face Bam files, Sam files, Cram files and annotate them with Bed files (with many different poorly documented standards) or GTF files depending on your Ensembl or UCSC references persuasion... Some file types are 0-based inclusive, some are 1 based exclusive, some call chromosome just ints (1,2, and then error when X, Y if you really assume they are ints) some call them chr1, chr2, chrX etc. We have fastq files with different quality encodings depending on their age, I recently had some strange errors, turns out the R2 of a fastq file is reverse complementary from the standard (why?!)... It's a mess and many people that spend years in the field still don't know more than just "choose one side and stick with it"... We also have a huge amount of frameworks for data analysis pipelines, Snakemake, Bpipe, Broad's WDL, etc, etc, etc. Not to mention the insance amounts of read mappers (BWA, RSEM, Tophat 1 and 2, Kallisto etc). Need I go on?
Thinks like [B|S|Cr]am files are complex in their own way, but mostly they're just kinda annoying to have to figure out. You can dig in as deep as you need pretty easily.
On the other hand, maybe I'm over-simplifying because I feel more comfortable in this domain, which I think is your point.
Btw, the files are ok and with effort can be understood and maybe some could even be considered elegant. It's just the shear amount of "standards" or rather the lack thereof.
With webdev I can at least stick with jquery and do no harm
The amount of time saved by doing this would be immense based on my experience inheriting maintenance projects in Anglaur / React / whatever flavour of the month framework was used. They may seem productive when developing, but trying to fix bugs in them can be an absolute nightmare.
And if you think Django is murky, try Spring :). Django is one of the easiest frameworks to work with simply because it includes so many functions you would otherwise need to somehow integrate. And even then there are still these issues.
Good post.
What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
Unfortunately, giving away a free resource like that doesn't balance the books and years of neglect later the site no longer represented modern Django. It's entirely possible the new Django book site is just as good, but the paywall discourages novices and experts alike. My friends who stayed webdevs think Two Scoops of Django is close in scope, and comes with free updates. It's not quite the everbook that I had hoped the original site would be, but is highly focused on best practices.
("2 Scoops of Django" is an excellent book to describe best practices. It's something I haven't seen the equivalent of elsewhere).
I just like producing cool stuff and don't want to be manually running SQL queries in both dev and prod.
I assure you it's not, not using frameworks can also be a purely pragmatic choice. If you deviate from their intended use, they can very quickly become an annoyance instead of a time gainer.
Django's documentation is great for developers who know their way around documentation like that as it is very comprehensive. It drills down nicely and is quick and easy to understand if you otherwise know what else is going on.
What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
No, most web developers aren't implementing the most complex and intricate algorithms or 'close to the metal' or whatever other gatekeeping people want to claim, but that doesn't mean it's easy work. I've never had a job where I didn't see some green engineer laugh off a feature request with "Oh, that's easy, it's just calling an API and putting the information into some HTML, what's so hard about that?" and then getting stuck when their first implementation fails QA, or the product manager explains that the customer needs it to be ever so slightly different, and the complexity of those edge cases turn out to not be so easy to deal with as the world of perfect computer science might suggest.
Is it hard in the same way? No, I will fully admit that coming up with a novel algorithmic solution requires a different mindset and brilliance that very few, if any, really good web developers I've met really have (although I'm certain there are people who excel at both, they're rare enough that I haven't been lucky enough to encounter them). But being really good at building a solid, secure, modern website isn't easy either, and it's also valuable.
This post focused on the complexity of the tooling, which yes, is definitely a thing, but if you don't know web development, and you know that you don't know web development, why are you attempting to build something using complex high level tools designed for and used by advanced web developers? Then again, if you don't know web development and you don't know what you don't know, and you've just heard 'Django is good' how would you know to maybe pick Flask or something simpler to understand?
I don't have an answer or a larger point other than this: writing software is difficult and complex and none of us should look down on anyone else because what they do seems easy on the outside.
You end up cobbling together half a dozen plugins (auth, forms, orm, admin) and adding dependencies that, albeit useful, are complex themselves: Celery, SQLAlchemy, etc.
The first Google search for hosting should be fine. There should be a simple tutorial for uploading your HTML file.
That's all there is to it. Beyond that, maybe you're just trying to bite off more than you can chew because you think you need things. It would be like going to ACME engineering supply and grabbing a bunch of stuff off the shelf because it sounds good. Then you get home and you have no idea what these things do.
A lot of this stuff looks deceptively like it all goes under the category of "web development." Sending an email isn't web development. That's dealing with a different protocol and different server software. You don't need to go far down that path before you start to wonder how any of this mess can work at all. I'm not even talking about web development, I feel this way all the way down to the metal. Yesterday I spent hours trying to troubleshoot an issue with Ryzen 3rd generation chips apparently having a problem when you load up all the slots with RAM (I threw in the towel because I can get by with half the RAM.)
That has always been the case. Software hasn't gotten worse, it has always been bad. But functional. Maybe it has become more complex.
Edit: Also 3600 with B450 board. Kingston DDR4-2666 Ram (8x4) which is all the computer store here in smallish city in the Philippines had in stock.
I used to work for a very large financial news company. They were all javascript. I though to my self: I learnt JS in 2008, this should be easy (it was 2014). Apparently very hard.
HTML isn't HTML anymore JS is forced everywhere, and everything has a different name.
Anyway, I disappeared back into HPC/VFX/machinelearning, to re-appear into webdev in 2020.
I was forced to do some front end work as part of a bootcamp. It took the best part of a week to make a bit of text a link.
The template library was so far removed from anything googleable, there was no documentation, worse still no style guide or example style guide.
I am glad I don't do web GUIs, they are hard and nasty.
You can of course use React with 2 different libraries to achieve the same goal, but why would you do that.
They need to rewrite it to use hooks at some point, but it's still a pretty good intro.
What is the reason that not using hooks is so frowned upon? When I took my first steps with React, I didn't know hooks, so I write component classes and it immediately "clicked" with me because it behaves like most other GUI frameworks do, from Win32-based (1) to X-based to Android to iOS to macOS to Java, and tons of higher-level frameworks in between. It also feels obvious to write a component class because you are describing the behavior of a stateful thing, that is, exactly what classes are made for.
With hooks, OTOH, we have a stateful "function", but all state is implicit, which describes behavior other than its main behavior (rendering) but that is implicit too... I can see why developers outside the React world don't "get" hooks, because I can't even remember seeing this idiom outside the React world. (BTW, I've seen the claim that hooks are easier to reason about, for which I found the opposite to be true, and the claim that this makes the main function stateless, which is absurd since the state just becomes implicit).
(1) the raw system-level APIs are often not class-based, but nearly all frameworks above it are.
edit: the footnote broke formatting
Hooks are actually much closer to what react does internally, and therefore behave much less surprisingly once you get into advanced things. Think of hooks like a missing primitive (similar to functions, classes and closures) that are critical to the react language, but don't actually exist in JavaScript and thus have to be approximated.
Can you give an example for that? So far, they worked pretty much as I expected. (Coming from a mostly Java background, which is relevant for what I expect).
As for why people like them so much as opposed to classes, my guess would be that they let people more easily write reusable behaviors for common tasks (e.g. look at this one for integrating with RxJS Observables for projects using RxJS https://github.com/LeetCode-OpenSource/rxjs-hooks#examples - with classes you would have to put code in several lifecycle methods, sometimes duplicate code, for each individual component).
Hooks also simplify working with React context a great deal. If that's your option for state management.
React is unnecessarily hard, and this fact probably keeps the UI development out of the hands of the people who could potentially do the best work (graphic designers).
If you want, there are APIs and frameworks that will allow programmers to configure print layouts and typography by hand.
As you can imagine, they're not as well known or as popular as the Adobe tools used for this work.
I'm not saying that web based UI development isn't without its unique challenges (state management), but I will say that things could stand to change, and we should move towards simplifying stacks, for what will ultimately be the user's sake.
I think this has a lot to do with it, interestingly. The web has become so ubiquitous, and the major players are so resourced, that the barrier to entry can seem really high. Maybe I'm wrong, but my perception is that potential customers wouldn't take a second glance at my sight if it were plain html with minimal CSS and didn't render perfectly on mobile.
> I am glad I don't do web GUIs
Same.
People seem to be ordering more of that, not less. Look at Reddit. They have all manner of stupid animations now.
This highlights the problem with web dev quite nicely. Not doubt you weren't really just making some text in to a link. I know the sort of thing bootcamps do, so you were probably installing and learning React, displaying a set of components, implementing React Router, and figuring out how to update app state based on the browser's History API. All with a template library that was a terrible choice for learning (because it didn't have docs). You could summarize that as "making a bit of text a link", but that's glossing over the details.
What you were really doing was learning component orchestration, reactive state management, and a browser API. Why wouldn't that take a week?
So the problem with web dev is actually that it's really, really easy to trivialize. You can make hard things sound like they should be very straightforward because they look a lot like something simple from the past. And when people find that they're not really doing the simple thing any more they complain because the new, harder way of doing something (which is actually doing something completely different eg reactive state management and in-browser routing vs asking the browser to do a GET) is hard.
"...component orchestration, reactive state management..."
Webdev tool makers are sadists, and many of the 'thought leaders' in the field have Stockholm syndrome.
Some aspects of web development certainly are hard (security especially).
Frontend development however, does not need to be as hard as it currently is. At the very least, I think there's an opportunity to simplify that layer of the stack, and give the control back to design professionals.
Dreamweaver to my mind, represented a more productive direction than React currently does.
Also, don't forget that the web is fantastically backwards compatible. You can write web apps without the complexity if you want to. You can use Dreamweaver! You can server-side render every page. It works. If that's what you want to serve your users then you should do that. Just don't complain when they use the competitor's app that's faster, more responsive, and does fun things like working offline.
The problem with current development trends is in how the complexity found in the tools, makes the pool of available creators (at a want for a better term) a tiny, exclusive club. This has a detrimental effect on efforts to innovate at design / pre-production stage, because the designers are near enough held hostage by us developers.
It also has an overall effect on project velocity, because work that could possibly be covered by techically competent designers (namely HTML/CSS development), can only be addressed by expensive and exclusive 'framework specialists'.
I strongly believe this can change by rethinking where the complexity lies.
The culture built around over-complicated tooling is possibly analogous to placing the responsibility for the production of a feature film in the hands of the VFX and lighting teams, or perhaps only deeming a mechanic as having the aptitude to drive a car.
I'm not rigidly stuck to this idea, I'm just musing here.
This analogy is a really good one, but you've got it slightly wrong. The website user is the person driving the car. The developer is the equivalent of the person building the car. Saying the development process is "too complicated" is like saying developers should only ever build compact family cars like a Ford, but some of us want to be building BMWs and Teslas and Porsches and F1 cars and monster trucks and semi-trucks, etc. Sure, it's more complex and difficult, but the end result suits the user better, fulfils their needs better, and often just makes them a lot happier.
That's a tiny minority of users. And that sort of user probably has an old, slow computer which is crushed under the weight of your vast javascript framework, anyway.
Whatever about a _better_ experience (some subjectivity there, though I don't really buy it) I'm pretty sure the modern web isn't a _faster_ experience for most users than it was a decade ago.
React isn't vast. React and ReactDOM together are about 40Kb over the wire of total download size if they're gzipped (https://bundlephobia.com/result?p=react@16.13.1 && https://bundlephobia.com/result?p=react-dom@16.13.1). For reference, the latest version of jQuery is 30Kb. Personally, I test all my production builds on a Chromebook that has 2GB of RAM, and they usually hit 60fps even on relatively complex pages. This is what the standard should be for building for the web - small, fast web apps that work well on slow computers and slow, high latency internet connections.
You can decide not to support everyone if you want, but that's a decision. The technology already works quite well if you put the effort in to building things properly. Build complexity is not a good excuse for serving slow apps.
It's possible in principle to build small fast SPAs, but they're vanishingly rare in real life. And even where code size is small, CPU usage and latency are often bizarrely high; even that 500ms latency user might find a roundtrip to the server faster than waiting on some SPAs.
I do a lot of my personal browsing on an elderly (six year old) iPad. It's totally fine for normal web pages, but every year the number of websites which are painful on it rises, due to (usually largely unnecessary) javascript.
There is an Icon, and some text that reads: "you are on a trail find out more" The find out more is a link, make sure all the text _and_ the icon are included in the link.
So its not creating a new state or widget. its literally moving a href tag.
Well it would have been, if it wasn't for the template engine, and what ever the state engine was/is.
Now, if this was QT creator, I'm pretty sure I could have done this task in about 4 hours (I don't know C++ very well and I've used QT about 3 times before, each for less than a day.)
As this is all templateable and composable. why in living fuck isnt there a GUI to do this sort of thing?
I mean, on a large website you need to make widgets, then you need to compose those widgets into screens. Those screens maintain state (basically flash, but with 100x worse performance)
the people that are styling the widgets are not always the same as those adding the logic, so why not make a tool where a visual artist can actually work visually?
So why on earth is it not drag and drop? Yes some things are auto generated based on context, but a lot of things are static screens where the widgets update the state.
Framer X. Sketch. Adobe Xd. etc
I mean, on a large website you need to make widgets, then you need to compose those widgets into screens.
Storybook (sort of).
So why on earth is it not drag and drop?
Dreamweaver.
The tools you want all exist. Those are just a few examples. There's hundreds more. You just didn't look very hard.
The main thing to remember is that you can only show the way - the person in question must actually step forwards and be motivated enough to go through the learning process. Otherwise - you need to think of another career. Programming is not for everyone IMHO.
Hang in there, there's a relative important chance that the fog will clear for you in some months/years.
The hard truth is, software engineering today is made of layers upon layers of abstractions. You have to dig threw those layers, understanding sufficiently the inherent and different complexity of each one before jumping to the next. And then finally it'll make sense, and new layers (there's a new layer every year btw) will be easier to grok.
If it helps, we've all had to go threw this learning process. All of us. However I don't think everyone is made for enjoying this process, that's another hard truth.
The real difficulty is sticking to the learning process while everything and everyone is screaming at you "Go! Build something! Ship Something! Show something you lazy b".
All the above are quick hacks that never really did the job properly in the first place. So they're propped up by a creaking pile of frameworks and other support systems that try to mitigate their awfulness, but mostly just add extra awfulness of their own.
Meanwhile backend systems - core server and db infrastructure, user authentication, computing resource allocation, payments, media and content management, privacy, and project build systems - are even more byzantine and non-standardised and more or less have to be retooled from scratch for every project - a huge job which duplicates an incredible amount of work across the entire industry.
As a corollary/reformulation of that sentence you get that what businesses are trying to do in the web is clearly something it wasn't made for. Yet businesses want to free-ride on the web's success, and are ruining it.
I've been using React in all of my projets since it came out in 2013. That's 7 years. React's (+ node for backend if needed) been one of the recommended for at least 4 years.
And lots of things have changed ! Using modules and building complex apps now with Webpack & co is so much easier than before. Not needing to support IE is also a blessing.
The piece doesn't say how mobile dev would be easier. Or how embedded dev would be easier.
It just says : dev is hard. Yes it is, it's a a full job.
Hopefully nocde solutions will reduce the gap for some use cases.
Still, when I glance at the JS frontend world people are still looking for the next hot frontend framework all the time, even if many are doing something similar to React. Going by github stars, Vue is the biggest one now. There's also Angular, Svelte, Ember, and Aurelia.
Unfortunately we're stuck with this Rube Goldberg collection of excrement, and it pays the bills. So there's nothing much you can do except lean in and embrace the suck. Maybe hit yourself in the forehead a few times with a clawhammer or start carrying a hipflask to keep yourself at happy-drunk.
I have also worked in programming for the sciences my entire career, multiple decades, would not want to do anything else. There is an awful lot of actual complexity out there especially in the life sciences which is inherently messy ... a system of exceptions ... Sometimes looking at the stuff that has to happen now, just to give information away for free it is staggering. The only conclusion I come to is complexity envy, Because I can't believe all this inevitablely became necessary. Yes there are bad actors and good security is very hard, and much more so if you are transacting money, but in other cases you can't really steal what is being given away but now there is the expectation that all this extra stuff is par for the course it you are putting on the web.
You have to read between the lines but what we have in this post is a Data Scientist skilled in the Python programming language struggling to acquire the shared knowledge required to build and deploy Web Apps. One toolchain (Python), one framework (Django), one backend architecture (12-Factor Apps) but the narrative surrounding URLs, HTTP, DNS, TLS, REST APIs, JSON, CORS, and www-forms is missing.
We seem to be missing a simple way to learn the shared knowledge needed to build apps in a world reshaped by Web, Mobile, and Cloud technologies.
I hope that after this crisis we will care less about making people click ads than about learning how stuff works in biology.
except maybe if status is a sufficient payment?
mayyybe good if altruistic ideas are more likely to propagate under those incentives?
That's the problem with web development today. Something that not only should be straightforward, but used to be straightforward, is now very complex. Partly because it involves connecting a large number of complex and constantly changing packages.
The last time I wanted to do something with an API, I coded up the API in Go and ran it from FCGI to get multiple copies. Easier than all those "frameworks". Probably faster, too, since Go is hard-compiled.
As someone who remembers having to do weird CSS tricks for really basic layouts (negative margins? hidden extra elements that don't make any sense, but cause other elements to move?), manually modifying the UI made up of jQuery plugins that require a very specific prescribed HTML structure to even work, or doing significant adjustments for 3 or 4 different browsers, I definitely dispute that claim.
Web development right now (at least the frontend for sure) feels like a breeze compared to about 10-15 years ago.
Steps 1 and 2 of the whole process may have gotten more difficult, but steps 5 through 100 are a lot easier these days.
It's possible to design coherent easy to use UI systems. Flutter and Qt are great and both much easier to develop for than the web IMO. But they don't have to worry about running in web browsers.
On one end of the spectrum, simply putting content on a website isn't so hard, right? On the other end, making a web application is understandably difficult.
Features like users, payment forms, and email confirmation (so e.g. the website can be used for a course) sounds like they ought to be as simple as the former, but are closer to the latter.
Is the value proposition from the website building websites like Wix that they take care of the hard stuff?
Perhaps things have to get complicated enough for it to get priority, perhaps cool things will happen all of a sudden. I was highly amused by the spreadsheet programming tool a while back. It seems a giant leap in the other direction.
The problem is the OP didn't test anything: He launched and users couldn't log in.
Ideally I'd like to gain 100% visibility into everything related to my tasks in webdev, but it's impractical and counter productive.
https://web.archive.org/web/20200507060013/https://jessimeki...
It's not static; it's a platform for helping people (life scientists) learn to code.
In general, Heroku has been a great platform. The fact that you can just `git push`, and your dyno rebuilds and restarts and your changes are live is so cool.
I mean, bioinformatics is _notorious_ for spending just enough time working on software to publish a result and no more. If we want to talk about rabbitholes I nominate BioPerl.
> here are a list of terms that you’d never come across as a bioinformatician: ["X-Forwarded-Proto", "proxying", "requests", "TLS connection", "HTTP", "loopback/localhost", "whitelisted proxy servers", "gunicorn", "nginx", "Apache/httpd", "HAProxy", "Django", "SECURE_SSL_REDIRECT", "SecurityMiddleware", "adapter", "URI"]
This list is more than a little disingenuous:
Django: This is like, the framework OP built their software on top of. I don't know how one gets the point of filing this bug without knowing the term Django
requests: literally, a user's web request.
HTTP/URI: this is the web developer equivalent of a biologist not knowing the term DNA. Or DOI for that matter.
whitelist: the opposite of a blacklist
Much of rest comes down to the simple fact that job of web developers is to build a network service. This is primarily done with layers; each layer only needs to know how to interface with the adjoining layers above and below, like a lego brick. This is actually a _solution_ to the complexity, allowing us to constrain the design of each layer to interact only with those above and below it. This layer abstraction lets us reason about each layer locally, independently of the system as a whole.
nginx, Apache/httpd, HAProxy: in this context, mentioned as possible proxies forming the outermost layer. in this context, accepts HTTP/HTTPS connections and makes HTTP connections to gunicorn
gunicorn: the thing that runs the Django app. Accepts HTTP connections, makes WSGI calls to the Django app
Django: Accepts WSGI function calls, routes them through middleware, then to the app as you define in urlpatterns
SecurityMiddleware: a class that potentially modifies inbound requests and outbound responses
Finally, OP is in the unhappy situation of bridging two network protocols: HTTP and email. The normal trick of using a protocol relative URI will not work here because you've gone outside the browser request-response loop. This puts them in the unhappy state of the lowest layer needing to know the details of the highest layer, hence the discussion of absolute uris. This is a direct violation of the local reasoning constraint, and what makes all of this hard.
X-Forwarded-Proto: An HTTP header, added to incoming HTTP requests designed to indicate to layers below which URI scheme was originally used, HTTP or HTTPS.
SECURE_SSL_REDIRECT: a setting for SecurityMiddleware that will stop processing the request and send a response to the browser telling it to resend the query over https (an HTTP 302 redirect), if the incoming connection is seen as coming over not http.
tl;dr: email is the thing that sucks.
Some software is legitimately hard to write, even some websites. But this isn't an example of that. The author has been sent on a wild goose chase.
CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness.
Because it’s a pretty generic e-commerce application that could have been built on the web 25 years ago. It should not have been complicated to do.