This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.
Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forking was generally considered a tragedy, something I managed to change five or six years later).
Rich Pixley came and said “all of our code should be in revision control, and I want to use this newfangled thing called CVS.” Michael was OK with it but John and I were steadfastly opposed. We agreed to an experiment, grudgingly, subject to a whole whiteboard of absurd conditions (“must be transparently integrated into emacs so we don’t have to know it’s there).
Pixley agreed to all of that and then ignored all of it completely. It was immediately such a win that everybody adopted it without complaint, including us two obstreperous a-holes.
A few years later a preventable crisis was how revision control first became client-server.
Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.
Some elements are inevitably going to end up being de-prioritized, and pushed further into the future. Features that do end up having a lot of demand could remain a priority.
I don't think this is even a case of "ask for forgiveness, not permission" (assuming you do intend to actually work on w/e particular demands if they end up actually continuing to demand it), but a natural product of triage.
Why, that sounds positively... Agile. In the genuine original sense.
It requires a level of trust and a level of performance, that I now believe doesn’t scale. Note that we had only a single junior developer, an intern, who wrote our bug tracking system but didn’t otherwise interact with the code base. The rest of the team consisted entirely of experienced senior developers, including management. I think at up to about 25 employees, the shortest working experience on the team (apart from the intern) might have been 10 years of shipping code.
But I really respect Rich for taking this approach which is why I referred to him by name.
Git is not a natural development at all. Obviously, it is a standard right now.
But I as a hobby coder at my teens I started out with FileZilla and copying over index2.php, index3.php, index_final.php, index_final_2.php and all of it worked well enough when at that point.
I took a little break from that hobby, and it still took me a lot of time to build intuition around Git when I started out professionally.
Obviously now that I have over a decade of professional experience I think it's a no brainer, but I don't think it's natural to understand it as it's suggested to you. I at least kind of had to go along with it and trust that it's good. It felt frustrating at first as many other things are. Git as a thing was FAR more difficult for me to understand in my 20s than PHP and how to do anything with it was for me at teens. Coding is raw logic, Git is about imagining teamwork at scale, which is a whole other thing.
The funny thing is that I feel like I was overall still more productive when I was at my teens building stuff solo with FileZilla and PHP, compared to a corporate environment now with all the processes and 1000s of engineers.
Git is actually an unnatural development. Its UI is atrocious. And it worked like crap on Windows for forever.
Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.
Then Github won due to VC cash dumping and foisted Git on all of us. Source control was no longer fine.
Thankfully, there is now Jujutsu(jj) which works on top of the Git storage layer so you can forget about the craptastic UI of Git. Source control is now fine again.
You can thank BitKeeper for all of this, and Andrew Tridgwell for forcing Linus Torvalds into creating git.
Gits internals can be confusing at first, but once you understand them there's really nothing you can't do. Being distributed by design also helps.
Plus it didn’t have a mechanism to send contributions. I think GitHub “won” because of its web-based PR system.
If you think SF+CVS is equivalent to GitHub+Git then you never used SF+CVS. Git won because of GitHub, specifically, not because generically it could be hosted on the Internet.
GitHub is to sourceforge what Facebook was to MySpace. MySpace was first but it was buggy as hell.
A more apt comparison would be comparing Facebook to a hypothetical social media site that when you click on a thumbnail of a user's image, you get a fullsize image of something like goatse...which thankfully doesn't exist(yet).
Nope, MySpace did. Or blogs, for that matter. Facebook just co-opted the phenomenon, and has for many become synonymous with it. Just like GitHub has managed to do with git.
Lots of companies were doing Git and Mercurial "forges" at the time. Many of them were better than Github.
Everything was evolving nicely (including Git and Github) until Github used VC money to offer everything for free to blow everybody out in order to lock people into their product (for example--export of anything other than code is still hit or miss on Github).
At which point, everything in the source control space completely collapsed.
You might want to go review your history before accusing someone else of that.
Github took $100 million from VCs in 2012. It then took another $250 million from VCs in 2015. Who the hell else was even within an order of magnitude of that in the same timeframe? Nobody. (Gitlab took almost the same amounts but did so somewhere between 5-8 years later depending upon how you count).
Bitbucket got bought by Atlassian in 2012. Atlassian was bootstrapped until it took $60 million in VC in 2010 and had revenues (not profits) of about $100 million in that time frame. It had nowhere near the resources to be able to drop the equivalent of $350 million on Bitbucket between 2012-2015.
VCs didn't make git successful, developers did.
In any case,that doesn't really matter considering that git had big players adopting before GitHub got any sizeable investment. And I'm not just talking about Linux. Rails migrated towards it when GitHub was still in beta.
> Bitbucket got bought by Atlassian in 2012.
29th September 2010[0] (just over a month after Atlassian raised the $60M.) ~18 months before Github took any VC money. If the VC money was key to Github's success, why did Atlassian/Bitbucket's 18 month head start not get them anywhere?
[0] https://web.archive.org/web/20160303204710/http://www.itwire...
Yep.
> github just makes things easier.
Which is the transition between steps 1 and 2 of Embrace, Extend, Extinguish. Though the last step is nowadays perhaps unnecessary, depending on how you look at it: Having co-opted FOSS software like git, what you need to extinguish is not that itself, just any other way to use it than your own. The main example of step 2 was of course "Pull Requests".
And I definitely think you're wrong. GitHub has managed to get itself conflated with git, and has benefitting hugely from this confusion.
Atomic commits can also easily be moved around (since the repository is a tree of commits with the commits being leaves), and they also make merging simpler in many scenarios.
Larry McVoy was famously wont to engage in trolling upon the Linux Kernel Mailing List, whereupon he did boast of his BitKeeper, which possessed atomic changesets. Concurrently, he did deride Subversion for its lack of the same. Thus, a great drama did ensue, one which ultimately bestowed upon us the creation of git.
git has also succeeded as a DVCS where others have faltered, for various reasons. For there have been monotone, darcs and other such systems; yet, it is chiefly git that has endured.
Nope. Subversion made them mainstream:
"An svn commit operation publishes changes to any number of files and directories as a single atomic transaction. In your working copy, you can change files' contents; create, delete, rename, and copy files and directories; and then commit a complete set of changes as an atomic transaction.
By atomic transaction, we mean simply this: either all of the changes happen in the repository, or none of them happens. Subversion tries to retain this atomicity in the face of program crashes, system crashes, network problems, and other users' actions." [0]
[0] https://svnbook.red-bean.com/en/1.5/svn.basic.in-action.html
Subversion definitely has atomic commits. That was it's major advance over CVS.
The major difference between svn and git/hg is svn assumes a centralised repository, whereas git/hg are designed to work as distributed repositories that exchange change sets.
Turns out that you can build sophisticated work flows on top of features they added to support that distributed model - things like PR's. These things are useful even if you have a single centralised monorepositiry of the style svn insists on.
But atomic commits aren't part distributed feature set. You need them in a monorepositiry. After you've experienced your CVS repository being turned into a mess by two people doing a commit at the same time that becomes obvious.
Well, that’s in the eye of the beholder. Yes, I hate the ‘program command’ syntax (“git add”, “git commit” etc) but I just call git-add etc and for me those commands are pretty clear.
But I understand how git works. I imagine most people treat it as a black box and then its behavior probably is rather obscure. I don’t think it was intended for that use case.
I acknowledge that Git is really good on a technical level, but don't like the CLI experience regardless.
Instead, I much prefer to use tools like GitKraken (paid), SourceTree (free, limited platform support), Git Cola (free, lightweight, a bit basic) and know that most of my colleagues just use the Git integration in their IDE of choice.
Working that way, it's a decidedly pleasant experience, not unlike using visual merge tools like Meld, where you can also stage individual lines/code blocks and operate with branches and tags in a very visual and obvious manner.
That said, sometimes the support for Git LFS and things like submodules is all over the place and I've had cases where not adding node_modules or something like that to .gitignore has made the UI unresponsive with how much stuff initially shows up in the working copy, so sometimes you still have to drop down to the CLI and do a fix herer and there.
> Well, that’s in the eye of the beholder. […]
I would say it is not.
Usability testing is a thing, and products can get better and worse scores when trying to do the same task / reach the same result. I'm not aware of any published studies on the topic, but I would not be surprised if Git got lower scores than Hg on UI/UX, even though they can basically do the same thing (distributed development).
Given the GP trained people on multiple version control systems, including another distributed one (hg), and people mostly had a problem with Git, I would say Git is the problem.
I only hear git complaints from people who don’t have an internal model that reflects what git is doing (I’ll agree that it uses a couple — but only a couple — of terms in a way that might violate some people’s intuitive model). But if you know what you want to accomplish the command line is pretty unremarkable.
But when you don’t know how the tool works you easily get snarled. And when you blindly resort to various “cheat sheet” sites it’s a crap shoot whether you “fix” your problem (how would you know?) or make things worse. That’s like me with photoshop.
I used to see this with CVS too
As stated up-thread:
> Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.
Seems that things only got hard(er) teaching people with Git.
> I only hear git complaints from people who don’t have an internal model that reflects what git is doing
Yeah, every time someone says that:
> git gets easier once you get the basic idea that branches are homeomorphic endofunctors mapping submanifolds of a Hilbert space.
* Isaac Wolkerstorfer, https://twitter.com/agnoster/status/44636629423497217
In the past I have used RCS, CVS, and Subversion. I never had to learn any kind of internal model for any of them.
Git itself is mediocre, and some other DVCS like Mercurial could have won out, although HgHub really doesn't have the same ring to it. The halo effect from the Linux kernel was also a factor. But the VC cash dump and Microsoft buyout came later, people used GitHub because it was better than the rest.
SourceForge had a messy interface and frankly I hated it. Google Code was like frozen in time, like most Google projects. Microsoft’s thing — I don’t even remember the name — felt half assed and like they were going to abandon it… and they abandoned it. There were also others… none of which I even think were serious enough.
Also Git won because SVN and CVS were centralized VCSes and you needed a server. I can just make Git repos before I even know if a project is serious.
There were other distributed VCSes of course, like Hg, but they either cost $ or wasn’t backed by a major user… like Linux. I admittedly just waited this one out and chose Git because it was more popular.
SourceSafe did get abandoned... because your source wasn't actually safe in it.
Its replacement, Team Foundation Server got rebranded as Azure DevOps; but the product's still around, 19 years later.
But the vast majority of people I personally see who complain about git simply don't know how to use it, and apparently don't care to learn. They say it's hard, and then they joke about never learning more than three git commands. Can't we do better? Is that where the bar is?
Git lovers mostly use it as a file copy service.
Most standard tools we use today are not obvious to beginners nor would they be the first thing that beginners reach for.
But senior developers can understand the problems that they claim to address, and why they are important and common problems.
I'm 100% sure that once I left that the devs went back to remote server crash and burn FTP development...they couldn't be bothered with the "hassle" and unneeded headaches of git.
By 1994 I had moved to other companies and a NeXT equivalent to svn with a good GUI - and then ClearCase.
Have you considered introducing Mercurial or even Subversion?
While Git may be a kind of an industry 'standard', if you're starting from zero, some of its concepts may be a bit mind-bending for folks, and it has often been commented that Hg seems to have a more beginner-friendly interface.
And if branching isn't going to be used (a large strength of git/hg), then Subversion may have an even simpler mental model (svn of course does branching, but the others are more optimized for it).
If folks are doing FTP-push deployment, then moving to 'just' SVN-push (commit) deployment can be an improvement.
Senior developers have simply grown accustomed to the warts, and accepted their fate - some even celebrating their mastery of the warts as necessary knowledge.
The only pragmatic reason not to change things is the compatibility chaos that will ensue and the resources required - not that problems don't exist.
Hi git fans!
Git is still a baby as these things go. Software version control systems go back to the early 1960s and descended from manual processes used to manage documents in large organizations, blueprints, and such from the early 20th century. Don’t imagine the Manhattan Project functioned by just passing around bundles of paper! There were staffs of people whose job was to manage document control! And I don’t mean “control” in the sense of secrecy, but making sure everybody was on the the same revision at the same time.
And for a field so steeped in building its own tools and automating human effort in development it is indeed astonishing revision control took so long to be accepted.
It only takes doing index73.php and index_old3.php for a few months and then eventually screwing up (unless you’re perfect) to realize how dumb putting numbers and _old4 at the end of names is. Then at that point, you naturally go look if there’s a better way.
That might be true now but it wasn’t always. Plenty of shops didn’t have any version control at all. And the build was whatever came off your local dev box. And “local dev box” wasn’t even a term…
That said, if I was going to start out teaching someone coding today, version control would be timestamped .zip files.
Oh no, we're in a different time zone.
>We'll put the timezone in.
Oh no, it's daylight savings time.
It's a false dichotomy. Before git there were other version managements systems which would've fit your use case much better than git. Subversion is the easiest perhaps.
When my consultant contract finally came to an end in 2003 they were still using it.
That isn't true, RCS was already in heavy use by 1991 in the GNU project.
I believe the problem was a combination of xenix (with limited length filenames) and sccs (which used 's.<filename>' to store each file).
Anyway, long filenames like longestpossiblename.c got checked into revision control as s.longestpossiblename where the s. prefix truncated the .c at the end
Later builds would fail since the checked out file longestpossiblename was no longer a c file and wouldn't compile.
Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow.
I left.
That's when I figured out what happened in 1998.
Looking at the http logs, around 5pm an ip address started spamming a whole bunch of requests to the same url. The same ip was doing pretty normal stuff on the website about an hour earlier.
My theory is holding down F5 key would cause the page to reload about 30 times a second. The website was not able to handle that many requests per second and it effectively became a denial of service attack.
This was around 2007-2010 and I think by now browsers have stopped repeating a reload if the F5 key is held down.
I worked briefly on a site that had this rankings page for users, and it was done by going player by player, and pulling the players table each loop to compare the current player to all the others. For things like "results against women"
Anyway you could DDoS the site by requesting that page. You could actually watch the page fill in as it computed, I want to say it was about a ten second load time.
DoS has become more of an issue now that all of the GPTBots are generating a lot more traffic. https://news.ycombinator.com/item?id=40865627
CVS was notorious for doing "text/binary" conversions (CR/LF line endings to CR and vice versa), sometimes inappropriately. More than once, it resulted in files where every other line was empty. I can very well see this happening several times, resulting in exponential growth of whitespaces.
But the seniors on the team pointed to CVS. Not the individual, not the "test" process, not the deploy checks.
Nobody noticed it slow cause everything ran on LAN. Only noticed the issue on 56k
The guy ate, slept, possibly got married while his finger was still accidentally holding down that key.
At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)
One company I joined was cleaning up the last vestiges of “customize it for every customer by letting people edit their copy on server,” which predictably turned into a mess. They were all very small customizations to styles or images but enough to make upgrades a total mess.
I did work at a company where despite having full source control they didn’t actually know of they could ever deploy the server component again. Edits got made to the live server but then made again in source control, or vice versa. There was one more senior person who couldn’t be talked out of their old workflow.
In theory everything matched.Eventually they even checked and got it all under control where they were positive it was the same and kept it that way.
But it had only ever been deployed from scratch… once. And for like 15 years it lived there and kept getting upgraded. It would all be moved when new hardware was brought in.
But it wasn’t installed from scratch. We truly did not know if we were capable of doing that. It is possible if that server was destroyed and we couldn’t restore from a back up it would take us an unknown amount of time. Even though in theory deploying should be as simple copying the files and starting the web server.
Were there odd configurations that had been made eight years ago that kept it running? Some strange permission changed somewhere?
I wasn’t on that team. But it always made me nervous. That was absolutely a core application of the business.
I really like small shops sometimes. You get a lot of freedom and get to have your hands in a lot of disciplines. You learn a lot of things, including things that’s should never be duplicated.
I’m not saying all the data in the database. Pretend your DB is fine or you have a full up to date backup. I know losing that KILLS businesses. But in my example just the web server immolated and the backup tapes sitting next to it melted.
Can you set up a new web server?
As a Java developer it’s usually install OS, add Apache + Tomcat, set a few basic config parameters, copy WARs.
PHP or Python is even easier.
They didn’t know they could successfully do that.
He had a spreadsheet where he kept track of the versions used by every customer. Once identified the version, he would open (no joke) a drawer in his desk and pick the right USB stick with that version on it.
I've always wondered whether this overhead was a worth price to pay for not wanting to learn a couple of SVN commands.
I said we already had source control (CVS) and a a working build process (a script on a server that pulled from CVS, built a jar file and deployed it to X servers). We were too "busy" at the moment but would look into it in the future. This never happened.
The real reason was that I was concerned that the central development center would take over the project, as they had tried to do so in the past. Looking back, I should probably have let them take over as there was more than enough work for everyone.
You need the “Dead Sea effect” to be in effect long enough that not only have the good people left, but for them to have been gone long enough that people rising into management have never even worked with somebody competent so they don’t know there’s a better way
1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS.
2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master.
The list goes on!
1. Previous consultants left no documentation or anything, and a running Hadoop cluster handling (live!) 300 credit card transactions a second. Management hired 8 junior sysadmins - who were all windows sysadmins, had never used Linux before, and were expected to take over running this Linux cluster immediately. They all looked at me white as ghosts when I brought up SSH prompt, that’s the point where I learned they were all windows sysadmins.
2. Another company: all Java and MySQL developers who were trying to use Spark on Hadoop, refusing to learn anything new they ended up coding a Java app that sat on a single node, with a mysql database on the same node, that “shelled out” to a single trivial hello-world type function running in spark, then did the rest of the computation in Java on the single node, management celebrated a huge success of their team now using “modern cluster computing” even though the 20 node cluster did basically nothing and was 99.99% idle. (And burning huge $ a month)
3. Another company: setup a cluster then was so desperate to use the cluster for everything installed monitoring on the cluster, so when the cluster went down, monitoring and all observability went down too
4. A Cassandra cluster run by junior sys-admins and queried by junior data scientists had this funny arms race where the data scientists did what was effectively “select * from *” for every query and the sysadmins noticing the cluster was slow, kept adding more nodes, rather than talk to each other things just oscillated back and forwards with costs spiralling out of control as more and more nodes were deployed
Any many more!
This might sound like I’m ragging on juniors a bit but that’s definitely not the case - most of these problems were caused by bad management being cheap and throwing these poor kids into the deep end with no guidance. I did my best to upskill them rapidly and I’m still friends with many of them today, even though it’s nearly 10 years later now,
Good times!
My favorite was at the company that was self hosting their code. The senior team lead wanted me to help him find a memory leak that plagued the product for months. Customers were told to restart the application every few weeks (this was a C++ application).
I sat down with the senior and looked at the code. I spotted the error.
I was like, "You know when you do new[] you need to use delete[]?" as all of his deletions were without [].
The look on his face was the best.
This seems like a pretty major lack of a specific piece of knowledge on the senior developers part, yes, but it seems like a much more unforgivable miss on the part of the code reviewers. Was the team stuck in a rut where only a single person (with coincidentally the same blind spot) was reviewing his code, or did multiple reviewers somehow miss this?
The fact that people study for exams has absolutely no correlation with how much they will remember or care for what they studied, none.
Perhaps you should have proposed anything-but-TFS (git/mercurial/even svn - no need for distributed in a small company... https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs...) but internal.
Might be a relatively easy sell if you just setup a local migration of their data to repository with a decent web gui for viewing the commits and associating with tickets, given my experiences with TFS in terms of slowness, and crashes/data corruption.
Haven't followed that for a while, but it used to be the case that it was Mercurial who was principled/fanatic about never rewriting history, ever, while git happily let you mess up with hard deletes.
What censor is for is the case where something absolutely has to be removed from the repo, for legal or security. It allows you to do it in a clean standard way, even replacing it with, say, a key signed statement, without forcing regeneration of the whole repository or altering the tree in any way.
Just remove the contents while keeping the hash.
They also do have the tools to completely rewrite history if you really really want to.
So, "principled fanatic" is not quite accurate I feel. They just have a lot more options than git (which also applies to their commandline tooling I feel, although of course there's a lot of 3rd party stuff out there for git to mimic much of what mercurial has out of the box these days).
I’d argue everything else becomes expensive quickly.
…but it is also imperative to remember any modification on checked out code is a branch, technically, regardless of the version control system used. This becomes important if your testing is expensive.
I joined a dynamic DNS provider once that had been around since 1999. Their tech, sadly, had not progressed much beyond that point. Showing the higher ups version control was like showing cavemen fire. Of course once the higher ups arranged to have training sessions led by the new hire for the entire dev team the VP of Engineering couldn't handle it and had me fired. Fun times.
That old project though was never fixed though, probably still running that way now.
I don't know if that made it better, I assume not much, VSS was really trash.
After we had off-site devs, the physical cap wouldn't work. So the project had a "silvercap.txt" file and you had to exclusively check out . And of course people forgot to release that and work ground to a halt.
Ultimately by the time I left I tried to introduce redbeanphp (orm), git for source control, and CakePHP for some structure. Nothing stuck. When I left it was still raw sql string queries, .zip files when they remembered for backups, and 400,000 line php files with everything caked on there.
For VB Sourcesafe was the go to tool if memory serves.
This is not a case of new vs old, rather incompetence vs competence.
I myself introduced SVN as versioning solution to a company in 2007, years after git was available I and I was aware of it
git simply didn't have the mind share jet, thus I went with what I knew
Git was too weird for a team, but svn with its linear timeline was a hit. They are still with svn, and for a small shop it's ok
In 2013, I was tasked with writing wrapper scripts around SVN to make it look like SCCS to avoid confusing the development people who only knew how to use SCCS whilst the company migrated to SVN. Fun but traumatising.
Well into the subversion era…
My attempt to introduce a "multissh" tool that automatically executed the same commands in each node at once was regarded with the highest suspicion and shot down. Shortly after I left, they had a multi-week outage caused by somebody fat-fingering the permissions on a network interface.
Some thought version control was an obvious improvement to make better, more bug-free software. Others had a fetish for doing things the hard way with little justification beyond emotional resistance.
Whereas nearly every UNIX installation included version control systems for free (SCCS in AT&T UNIX SVR1-4, RCS in BSD UNIX or both) that worked exactly the same everywhere.
Then you have to start tracking the limitations and issues in 2-3 year old releases and the solution to issues is never “upgrade to the latest”.
At that point, source control starts feeling worthless.
Source code management back in the days of VB (long before VB.Net) was not the same as what you see today.