My God, it's full of yaks
ronjeffries.com
ronjeffries.com
Aside from some expected path issues (builds too stupid to actually look rather than assuming a disk layout, etc) and some BSD <-> Linux issues (builds that assume all the world runs linux), I've had very little problem getting most things working on OS X.
It does help to learn what Apple has, uh, improved. This point is related to what an awful time I have getting anything to work on Windows - I hate it and don't want to learn about it.
At least in the article, there are hints that the author hasn't taken the time to learn about the platform they want to run code on. Which is fine - I feel that way about Windows[1]. But it is also a good idea to understand why you're having the problems you're having.
[1] Which is why I don't touch Win at work and the VM I run at home is a hacky mess that works for what it does, but only after a few aircraft carriers-worth of cursing and slash-and-burn hacking that I know can't be the 'right' windows-land way to do things. (The abomination is completely firewalled off from the outside world - I try not to be a disease vector.)
Having built applications for all three platforms I find they all require a certain amount of patience but once you understand the motions, it's relatively painless.
Totally agree. We had developers recently join our team, which primarily uses OS X and Linux. They were excited about using Windows since they had previously been developing in .Net. They now appreciate that development in Windows is not as easy as Microsoft wants you to believe.
The problem that won't go away anytime soon is that Microsoft spent years not making it easy on developers to develop in anything that isn't typical to develop in Visual Studio. They will not overcome that overnight. When a development team has focused on Windows, all is fine, but don't expect things to work without problems otherwise.
Maybe if Apple maintained their own ports tree, but they don't. There's no UNIX under there. There is a BSD kernel, a bastardized UNIX-ish base on top of an ancient file system (yes HFS+ has finally been replaced, but just this year) with a proprietary display system and yaks fucking turtles all the way down.
MacOS is a terrible developer OS. 10.6 and Expose and work spaces. 10.7 took this wonderful thing and replaces it with Mission Control; which only allowed one row of workspaces, no columns and windows were now grouped together so you couldn't see shit. No way to revert to the old behavior either.
Then Apple destroyed Final Cut. Goodbye video editing.
I switched back to Gentoo + i3 and never looked back.
I know there are now tiling window managers for MacOS. That's cool. I might try them one day. If I'm forced to.
Use vagrant or Docker or similar on the Mac -- this way you get reproducibility, snapshots, etc.
I'd do that even if I used Linux (in fact I used to use Linux and did just that).
Why would I want to setup my Desktop Linux box as my deployment/developer environment and have to change it and mess with it anytime I work on a new project / another deployment target?
>Then Apple destroyed Final Cut. Goodbye video editing.
As a professional video editor, I have to disagree with this too. They cut some features from FCP7 for the first FCPX release, but they brought them back, with more features and better implementations in subsequent releases. And all that with a codebase that's not some 2000's relic, but built for today's challenges.
Besides, I don't see what's much better to use for video editing on Linux.
A modern package manager, such as Nix or even most language-specific package managers, will allow you to set up some sort of sandbox for the project without having to take the performance hit of full-blown virtualization while developing. Of course, you'll probably want to spin up a VM while testing, just to be sure. And Nix can handle that for you too!
All while staying completely reproducible from your config files, all without having to bother with your virtualization program's snapshot system.
It is a little, but I've found that running headless (and no reason not to) drops this overhead considerably. And with 4 cores, 16GB memory and SSD, I hardly notice it.
Also modern containers like Dockers can drop the overhead to almost zero -- though I haven't really tried them to see how comfortable they are to work with.
Besides, if you natively run e.g. node, postgres, some work queue, etc, you're still gonna get the overhead on your host from those processes.
Nix sounds nice, but as you say you still want to spin some VM to know it all works in the final deployment setup. Not sure about your last suggestion "Nix can handle that for you too", will have to check Nix out more.
Docker makes this workflow a lot easier than multiple VMs. I'm running ubuntu with vagrant on macOS, and each project has a docker-compose file that spins up all the services. Stopping everything for one project and getting another one up is just one command, and takes a few seconds at most. With docker's layered fs, there is almost no space overhead.
Even if you don't use docker for deployment, you can just use a container as a psuedo-VM to use this workflow.
What do you use for video editing on Linux (that's in the same calibre of Final Cut Adobe Premiere)?
The video-editing software and support is vastly better under OS X than under Linux.
Nowadays, though, I run them on Vagrant (again on the mac) -- mainly to have snapshots and easy copying/reproduction of any target system. And even if I run Linux, I'd still run Vagrant on top of it for development too.
In my experience, it's far preferable to work with an isolated and repeatable process whenever possible. I really love Vagrant for this, since it scales from simple inline shell scripts to setup a VM, up to integration with your favorite flavor of provisioning software (Chef, Puppet, Ansible, etc.). I've setup environments that ran common provisioning toolchains that scaled from individual dev setups straight to production clusters. Similar motives drive a number of other approaches, esp. container-based systems like Docker.
Once you're fluent in these kinds of tools, you never really want to go back...
Ubuntu saves me so much time. (Also for some reason, Windows hates my graphics card and produces terrible sound in my BT headphones. Somehow a multi-billion company can't figure out BT while an open-source project can)
It's actually kind of revolutionary.
I have Windows 10 installed on a separate drive for when I want to play games but even then, I have to cater to Windows. Make sure to save constantly because sometimes Windows decides it is too much and causes my graphics card to flip. And I also have to plug my headphones in or else I get static-y mono sound.
Admittedly I could probably go buy "For Windows" headphones and figure out what is going wrong with the graphics card... but I shouldn't have to. We are talking about the latest operating system whose ancestors have been dominant across the world for decades, developed by one of the largest companies in the world. How are they getting it wrong?
You're also talking about them being able to perfectly support every modern piece of USB/PCI-e/etc connected hardware, which is no small feat even for the market leader. For comparison, I've never had these problems you describe since installing Windows 10. How can you be sure your success with Ubuntu isn't just good luck?
I've been doing sysadmin, programming, devops, etc professionally since 1995 and I've had to shave SunOS yaks, HP/UX yaks, Solaris yaks, Windows yaks, MacOS yaks, MacOS X yaks, Python yaks, Ruby yaks, Java yaks, Make yaks, C++ yaks, git yaks, cvs yaks, rcs yaks, ext2,3,4 yaks, Netapp yaks, Fibrechannel yaks, Cisco yaks, F5 yaks, nfs yaks, smb yaks, frame relay yaks (!), tcp yaks, rpm yaks, deb yaks, homebrew yaks, npm yaks, pip yaks, yum yaks, apache yaks, http yaks, xml yaks, tia568a no wait b yaks, serial crossover cable yaks, who the hell thought they could use 75 ohm as thicknet cable yaks, ... you get the idea.
We deal with a complex ecosystem with even more complex interactions that is often poorly documented at best. We're lucky when we can treat it as a black box and it all works. But often something breaks and suddenly we're hoisted by our own petard and there's yak hair friggen everywhere. I consider it a modern miracle that I don't have to pin ethernet duplex anymore! Glad we finally figured that one out.
Converse to your assertion, I switched to OS X back in 10.2 days because I got tired of shaving the hairball that was Linux desktop at the time. Oh, you want to print something? Here's an lpd yak for you... have fun! Oh you have a new monitor, here's some scanline settings to put in your xconfig that may work. Mac OS X is amazingly smooth as a desktop OS. [1]
But since you mention OS X, I'm reminded of of my favorite Apple technical note ever which began: ”As if it were a swarm of bees, you should stay away from the SyncServices folder in Mac OS X.” And that should apply to our whole industry, really... a swarm of bees.
Anywho, shaving yaks, story of my life, at least it pays well.
Pro-tip: when shaving yaks, take notes. I guarantee you you'll shave this yak again and it's handy to have your own reference to refer back to... even in these days of google, stack overflow, and quickly moving technologies. At worst, maybe it will be inspiration for a fun blog post some day. :-)
[1] So here's a fun story. Back in 2003 or so, I wrote a perl script which would forward emails to my phone via SMS. It did so by POSTing to a web form on a Verizon server. Worked fine when I was developing it on my OS X desktop, but failed to work on my Linux server. Yak shaving time. Eventually I'd broken out tcpdump after diagnosing that the Linux server, sitting on the network right next to the Mac, couldn't even connect to the Verizon server. Turned out the Linux server had ECN enabled, which being new at the time, Verizon's firewall was rejecting. Only had to peel about 5 layers of the onion on that one.
I suggest you try linux again, find new yaks and update your arguments. You haven't had to mess with scanlines in over a decade. I've never done it. Heck, you don't even need to create an xorg.conf any more. And some time Real Soon™ even X11 will be gone.
I remember editing XF86Config, .fvwmrc, and PPP init scripts back in the long ago, but I'm damned if I could do so from memory today.
This is “yak shaving”: http://catb.org/jargon/html/Y/yak-shaving.html https://joi.ito.com/weblog/2005/03/05/yak-shaving.html http://sethgodin.typepad.com/seths_blog/2005/03/dont_shave_t...
Use vagrant or Docker on the Mac -- you get reproducibility, snapshots, an environment fully compliant with any server you want to deploy to, etc, and you still get to code for it (editing source etc) from the Mac.
growing up at with a dad from DEC I actually learned how to use systems I've never used since :D but my dad ran Windows and hated macs (he worked at Xerox before DEC). so I grew up with dos and Windows for many years, then I switched to Linux when I was a teen.
I hated on macs and Windows (mostly macs) then, it was a shit show to my eyes and Linux was the ultimate (even though back then with my current pov Linux was the actual shit show, things broke and had to be fixed relentlessly)
I worked in more corporate type jobs for a while and used my Linux and Windows skills when needed, and then started in the OSS startup world and saw that everyone had a Mac, writing free software on a Mac, my head nearly exploded!
but I was given a shiny new MacBook Pro and so I had to give it a go. I was determined to hate it but once I found tools that let me use the command line I was fine with it. it was pretty slick, minus all the crap I had to do to get it working.
in the end I found they all have individual uses that the others aren't as good at and I use all 3 operating systems daily. the Windows boxes are good for some applications that just suck in wine, the Mac is good for applications that are written exclusively for macs but are very useful and I love that it turns on and is usable right when I open the lid (probably the only reason I bring it out with me when I leave my command center)
technology has gotten to an amazing point, especially if you're proficient enough to use and understand it (which admittedly isn't easy to do, but worth it if you can)
some of this stuff is just amazing, I run X11 on everything and I have a few very beefy servers that I use to just run my software on and display it on my main workstation.
using some newer tools I am able to use the video cards on my headless machines, I can run wine server on a dedicated wine box and run rdm or xencenter and when I decide to go mobile I just switch those sessions over to my Mac and use them there.
I have a IDE server that I run IntelliJ and all my other coding projects off of throug X11, it's got super redundancy and is more securely protected than my other boxes.
using automation tools like Jenkins, tasker, pythonista, auto input, auto remote, and Pagerduty I can get notified of any incident and automatically have the servers connected through dbus and yakuake send output to my phone with stalled processes etc..
its all good IMO
Getting a dev environment set up with nix is the usual amount of hassle. The difference is that nothing ever works by accident. Therefore, once it does work, I know I've captured all the dependencies and configuration, and I can reproduce the environment wherever I want.
For example, does nix play nice with CMake or config files? ie. I can just point them at the packages in /nix/ and it'll work?
Or for example, I often find myself wanting to have a few different version of OpenCV around (with different build flags) - but all version 3.0. Nix seems to make a new hash/folder when the dependencies change. Can you make make a new build/folder when the dependencies stay the same? (or specify to NOT make a new folder and to overwrite the previous compiled version - ex: you fix a bug)
I have a ton of overlapping dependencies between contracts and projects and this would be a cool tool to keep it all organized (right now in my workflow each project/contract would have it's own copy of OpenCV.. which is a bit of a mess)
(I'm totally cheering for this - hope it becomes mainstream, just wondering how "prime-time" is it)
It does sound like nix is a good fit for your use case. You could certainly create separate environments each with a slightly different build of OpenCV. They'd share dependencies where possible, but not step on each other where they differ.
Nix provides a tool called the standard environment (stdenv), which makes compiling classical unix software pretty easy. It assumes the typical download, configure, compile, install sequence and provides the right environment to make that work. It's really flexible, with lots of hooks that let you override the default behaviour and customize the build. The Nix manual has a pretty good overview of it. https://nixos.org/nixpkgs/manual/#chap-stdenv
Any change will lead to a new output folder, there is no option of reusing them.
Why not just give it a try in VirtualBox to see for yourself how far along it is?
Luckily there's a NixOS build cluster, so you don't normally need to recompile glibc yourself. :)
This is such an important property.
Usually faster than a couple days of internet searching.
Shave the yak once.
Write a script that shaves the yak for you next time.
Way better than Linux on the desktop. Way easier than configuring a second operating system to run the stack. I'm not going to production on MacOS, why would I develop on it?
Ironically, just watched someone whine about open source projects using Docker for dev on the 19th and then praising Docker for basically the same thing on the 22nd. Docker isn't magic and no one says it is and people yelling against that strawman are just making themselves look ignorant.
I use this framework for Python: https://github.com/agostodev/substrate
I use this Clojure template for Java: https://github.com/nickbauman/cljgae-template
But I think it doesn't really need to be that complex and involve so much yak-shaving. Around 10 years ago, when I wanted a personal web app of the similar CRUD type, I opened a text editor, browser, and the PHP documentation, and it was done in a few hours. Uploading a single file to the server was all that I needed to get it working and test it.
I thought I’d write this article about my experience, to try to draw some learning from it. I created the folder and the file, typed in the YAML and the first paragraph, and started Jekyll to build the html as usual.
Would it have been easier and faster to just write the HTML directly and upload it to the server instead of requiring an elaborate (despite hidden behind one command) "build" process that again depends on a large infrastructure of software?
Me, I just want to write programs, mostly small, that do things that may be worth writing about. And I want to write articles like this one, convert them to HTML, add them to a few indexes, and put them on my site
My advice is to avoid all the complexity and big systems that claim to do just about everything while requiring plenty of configuration and dependencies, if all you need is something that could be much simpler. Otherwise you are "making more yaks to shave".
But making a self-contained build is hard, so nobody ever does it.
Java has got usefully close in recent years. You can download and build a Gradle project with only a JDK and a shell installed, plus HTTP out to the internet; the project can (should!) contain a standard wrapper script which can download the right version of the build tool, and the build tool can then download the right versions of the dependencies and do the build. The downloads are cached locally so you don't download the internet on every build. The wrapper script only does straightforward stuff, so it doesn't require a particular shell or version, and the JVM changes slowly and safely enough that you can get away with being pretty sloppy about versions (ie any Java 8, 2014 - 2017+, will do).
There's no dependency on having some SDK installed, or some version of the runtime, or some build tool, or some package manager, or any other bits and bobs. Just a shell and a JDK. You don't have the problems listed in the article, and you don't have to invest time in avoiding them.
The next step would be for the wrapper to download the JDK itself, but i think that's unlikely to happen, due to bandwidth and licensing.
Once you're in the build tool, there are strong conventions about what's where. The 'build' task should do everything; 'check' should run every kind of validation possible; 'assemble' should build every artifact. Ideally, none of them should require any setup, although that requires some discipline on the developer's part.
I don't know of any good way to set up things MySQL or RabbitMQ if you need those for integration tests, which is definitely a problem. Perhaps we need a Docker plugin, so in my build i can say "to run the integration tests, download and start the following images, and inject the container IP addresses into the tests like so". The usual way to get around this is to use pure Java in-memory options for integration tests (eg H2 for a database, maybe something like HornetQ for a message queue), and you can get those like any other dependency, with no manual setup needed.
Sorry for the ramble. tl;dr yaks are bad, but there is hope for canned pre-shaved yaks.
When I set it up starting from Ubuntu packages, the process was still convoluted and involved bootstrapping from an older version of ghc.
I haven't tried it on Windows.
The download instructions say it depends on libffi, libgmp and zlib, which I didn't realise, plus some other tools (gcc, etc.) which I assume are for building packages that wrap C libraries.
http://docs.haskellstack.org/en/stable/install_and_upgrade/#...
What about getting the stack tool, though? Is it in the package managers at a useful version? It doesn't seem to be in MacPorts. Does it change quickly enough that you need to ensure you have the right version? Ah, i see that the installation process is:
curl -sSL https://get.haskellstack.org/ | sh
Perhaps each project should include a shell script which does this!
curl | sudo sh is much more of a problem.
True unit tests though ought be runnable outside GAE since your code should not be coupled with GAE if it is unit testable. You still need to test your code correctly integrates with google app engine.
I would take a look at tox and py.test for testing in python, these are what I use in my normal linux environment.
https://cloud.google.com/appengine/docs/python/tools/localun...
Here's the Java stubs:
https://cloud.google.com/appengine/docs/java/tools/localunit...
I wrap the hell out of the java stuff on App Engine because Java is so pedantic. It ends up looking like this:
https://github.com/nickbauman/cljgae-template/blob/master/re...
I would like to point to the Rust developers as ones fostering a more deliberate culture with tools and practices that manage change without exploding downstream.
We use OS X for development, for the most part, but Linux works well too.
We use nose2 with the following plugin to run unit tests: https://github.com/udacity/nose2-gae . It assumes your GAE SDK is installed at /usr/local/google_appengine.
For API tests, we generally create a webapp2.WSGIApplication object during setup, configured with the routes we want to test, and we send webapp2.Request objects to that application using request.get_response(wsgi_app).
Hope that helps!
* git clone
* setup.{bat, sh)
* Open IDE of choice.
* Click build button. (or run make)
The only dependencies are git, a default PC OS install (Linux, Windows, Mac), and your IDE of choice. Any project or system that can't provide that easily is too complicated by far, our non-technical CEO and sales staff can build and run our code without any help.
A quick search suggests webtest is part of Pylons[2]. But Pylons isn't listed as a built-in third-party library[3]. So I'm not sure if there's anything to fix.
1. https://github.com/GoogleCloudPlatform/python-docs-samples/b... 2. http://docs.pylonsproject.org/projects/webtest/en/latest/ 3. https://cloud.google.com/appengine/docs/python/tools/built-i...
In the authors case, webtest should have been installed separately, rather than using it from within the cherrypy source code, which seems to vendor webtest.
"Some further very simple tweak, which I no longer remember, and the test ran … and did nothing. That’s not too surprising, I guess, because there’s no
if __name__ == '__main__':
"You’re supposed to know that!"
...is also the fellow who had some trouble with Sudoku[1].
I say this not as an attack on Mr. Jeffries, but as a way of pointing out that shaving yaks is easier if you know to start with shears and not a hatchet.
And the article spends more bytes talking about whether some code is YAGNI, than the amount of bytes in the code itself?
>To eliminate this warning, please install libyaml and reinstall your ruby.
This line should make you think of using brew or apt-get or whatever to grab libyaml first. A few clues, the lib prefix is much more common for native libraries rather than interpereted ones. Second, the request to re-install Ruby means that this is a dependency that is required at compile time for Ruby, which rules out anything you'd install as a gem.
But maybe I just have shaved too many yaks and know what what already.
I did an app for a school project and was using Flask. At the time App Engine only had webapp and web2py available as frameworks (maybe it's different now) so a bit of shoehorning was necessary to get Flask working, and Flask was relatively young at the time so there weren't a lot of examples out there of the two technologies being used together. I recall having a lot of trouble working with GAE's file upload API.
The instructions wanted me to download a binary tarball of Go and untar it directly into /usr/local. As if I would never need to uninstall it. Apparently even though the version of Go has changed several times since Ubuntu packaged it, they think this will be the last version ever, or something?
I ended up hunting for an Ubuntu PPA that had it instead. Of course all the accepted answers on Stack Overflow point to PPAs that don't exist anymore. A comment sitting at some tragically low score saying "hey, try this one instead?" pointed me to something that worked.
I think you dealt with it the first time and you've gotten used to it. (And maybe there's some nice way to get a newer version of Go when you already have Go.)
BTW, you can put the Go SDK anywhere you like, as long as you set GOROOT. See "custom location" on https://golang.org/doc/install
The directions link to Stack Overflow, though I don't know how the search result picked out these steps, as Stack Overflow has a different accepted answer that doesn't work. Intervention by Google, possibly?
I didn't realize it would only make one directory. That is reassuring. I would still prefer packages as a way to install, upgrade, and remove Go, instead of remembering months from now that Go is installed differently from everything else and there's this directory I need to mess with.
Now, that all would have been more intuitive as something to do in my home directory. If it only needs one directory and it can go anywhere, why would these Google-blessed directions tell me to mess with /usr/local? I believe you, I just assumed that people would be averse to messing with system files without a package if there were any other option.
(I think of Python, where the one thing you're supposed to do with your system Python is use virtualenv to install a non-system Python. Which is another example of something that's only clear to veterans of the language.)
Now... this is a problem with a whole lot of programming languages. Programming languages change. Their libraries change way too fast for an OS-approved package manager to keep up, so you need a package manager for your language. And that changes too, and nobody designs their packaging system to smoothly update to a better packaging system designed by someone else.
Maybe one day every programming language and OS will unite under one purely functional packaging system and never replace it... haha, no.
[0] https://github.com/search?q=yak&type=Code&utf8=%E2%9C%93
Specifically, I spent hours and hours trying to fix
import matplotlib.pyplot
ImportError: cannot import name _thread
A ton of googling and swearing produced the fact that a package named six that does god knows what was being read out of
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six.pyc
/Library/Python/2.7/site-packages
ps -- 6 months ago the above used to work. Why did it stop? I have no bloody idea. But this consumed multiple hours of my life.
Install a virtualenv, and on OS X, maybe install Python from python.org, and then everything works amazingly.
Actually acquiring virtualenv on OS X seems not to be super straightforward, especially if you want to use the system Python interpreter. The best option seems to be to `brew install python`, and then `pip install virtualenv`. Or download Python from python.org, and maybe get Python 3 while you're at it.
Thus, I use my Mac as a fronted with business apps but all dev is done on Linux VMs. I like the separation and avoid all that hair pulling.
Now, dealing with proprietary HW is a whole other can of worms. I spent 3hrs last night trying to load a LE cert into a Polycom phone but had ZERO success getting the phone to use it instead of the factory cert. ARGH.
You have root, and you can build your own enclave of open-source code doing as you desire, but this is true of pretty much any system--certainly it's true on Windows, at least.
Linux is many (good) things, but up until this point in time, "convenience" is not a word I would associate with it. macOS and Windows own that world.