Harlequin: SQL IDE for Your Terminal
github.com
github.com
A few questions ...
a. what caused you to start working this?
b. any particular reason why you chose to implement it the way you did (e.g. I see you use Python + Textual as opposed to something like https://charm.sh/libs/)
c. any major functionality you feel it's missing?
d. any limitations (e.g. doesn't work with Oracle?)
e. any reason why someone should not use it?
Thanks in advance.
There was a Show HN: from the author and this thread and discussion last year:
Harlequin: DuckDB IDE for the terminal - https://news.ycombinator.com/item?id=37588526 - Sept 2023 (58 comments)
harlequin is also featured there as a tool of the week on that site.
Especially if it's a nice TUI app like this looks
EDIT: Guys, in my experience, virtual environments do not fix this problem in all cases. At least, not sufficiently for me (after getting used to Nix's guarantees, for example). Not to mention, there's multiple ways/attempts at creating and working with virtual environments: https://twitter.com/pmarreck/status/1735363908515295253). See below comment.
I suppose I could get around that by installing a Python interpreter outside of brew, and only use that for packages.
You can run pip in an isolated virtual environment, in user mode (not as root) - see https://gist.github.com/saurabhshri/46e4069164b87a708b39d947...
https://gist.github.com/saurabhshri/46e4069164b87a708b39d947...
python -m venv ~/my-venvs/<name>
source ~/my-venvs/name/bin/activate
pip install <package>
but stuff like this would usually get installed globally, and your projects would instead have a venv.personally my favorite tool is pyenv, which allows you to have many versions of python on your machine as well as many virtualenv (which are assigned to any version you have installed)
then it is as simple as
pyenv virtualenv <version> <name>
pyenv virtualenv 3.11.1 xyz
and to activate pyenv activate xyz
this allows you to keep every project you have isolated not only to the packages required to run it, but also the python version required.I work on a handful of projects that run on 3.10, 3.11 and 3.12. Each has their own independent python version, and within that version they also have their own python packages (pip environment).
At the end of the day these are simply directories on disk.
It's like ye-olden days. Now we can go back to static linking and bundling everything in jar files and call it a win!
but also...
> but stuff like this would usually get installed globally
well, you just killed (apparently unknowingly?) your whole argument right there, because globals are bad and absolutely not project-specific and absolutely do cause compatibility issues between different Python projects
If you ever come around to Nix, it takes care of this problem for good (as in, it guarantees that you will never have 2 projects that step on each other), across every ecosystem, not just Python's. Unfortunately, I don't see very many Python projects at all that contain a flake.nix file, which is a damn shame, because it would cause people like me to hate Python just a tiny bit less
It used to be hard. A global site-packages or dist-packages dir is hard. Conda and all those tools make it worse.
A virtualenv is a directory. It contains a copy of Python and everything needed.
You don't even need to activate it. You can simply use the binaries that exist in those directories.
It's literally this simple:
$ python -m venv pmarreck
$ pmarreck/bin/python --version
Python 3.10.12
$ pmarreck/bin/pip install harlequin
Collecting harlequin
Downloading harlequin-1.8.0-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 1.2 MB/s eta 0:00:00
...
Successfully installed MarkupSafe-2.1.3 click-8.1.7 duckdb-0.9.2 harlequin-1.8.0 jinja2-3.1.2 linkify-it-py-2.0.2 markdown-it-py-3.0.0 mdit-py-plugins-0.4.0 mdurl-0.1.2 numpy-1.26.3 platformdirs-3.11.0 prompt_toolkit-3.0.36 pyarrow-14.0.2 pygments-2.17.2 pyperclip-1.8.2 questionary-2.0.1 rich-13.7.0 rich-click-1.7.3 shandy-sqlfmt-0.21.1 textual-0.46.0 textual-fastdatatable-0.5.0 textual-textarea-0.9.5 tomli-2.0.1 tomlkit-0.12.3 tqdm-4.66.1 tree-sitter-0.20.4 tree_sitter_languages-1.9.1 typing-extensions-4.9.0 uc-micro-py-1.0.2 wcwidth-0.2.12
$ pmarreck/bin/harlequin
Boom, the tool launched immediately. No conflicts with anything else. When I am done, `rm -rf ~/pmarreck`, done.I did all of this in my home directory.
It literally cannot get easier than this.
You could do `for line in requirements.txt, pip install <line>`. They are ostensibly the same. It is not a magical lockfile. It is unix. It is just a list if packages. If you are in a virtualenv, you will be fine.
either activate the venv,
source venv_directory/bin/activate
pip install -r requirements.txt
or, skip the convenience and use the binary directly: venv_directory/bin/pip install -r requirements.txt
virtualenv activation just sets the $PATH to refer to those binaries. you can use them directly.the production deployment for my core python app lives in /srv/app, the venv lives in /srv/venv
To update packages on the system, it is as simple as
cd /srv/app
/srv/venv/bin/pip install -r requirements.txt
Then to invoke this with the correct runtime, it is as easy as /srv/venv/bin/gunicorn ...
In this example I am running the gunicorn application server. This is running the specific gunicorn version installed into my virtualenv.The name `/srv/venv` is my decision. You can call that whatever you want and put it wherever you want. For instance, if you have two projects called application-foo and application-bar, you can have the following:
/srv/application-foo/app - the codebase (ie, the github repo)
/srv/application-foo/venv - the corresponding virtualenv
/srv/application-bar/app
/srv/application-bar/venv
Some people will even put their venv dir inside of their source tree and exclude it from git (add to .gitignore), but I do not like this approach because my deployments destroy the app dir and unzip the build into that location on each deploy.I cannot speak for every python-based project (distinction from pip package) out there. A lot of people do not know what they are doing, open source is literally all the rope to hang yourself with. Anything is possible, and people without engineering experience will glue things together without understanding how they work.
If you are installing something via pip, then yes, you can create N virtualenvs and use them however you want. They are 100% isolated environments.
If you are using homebrew, apt-get, dnf/yum, arch etc... then those are going to obviously vary from distro to distro and that is outside the scope of this discussion.
I try to stick to Python's native tools as much as possible. Using a distribution package is going to cause issues, for sure. IE, don't install `apt-get install python-pil`, use a virtual environment and `pip install PIL`
OK, how would I include all of these under the same PATH regime?
So for example say I want to run this project from a commandline location elsewhere... I'd only be able to have one venv activated at the same time in the same session, right?
I guess that's part of my issue with this. I want to be able to access 10 different Python projects' commands from the same command line at any time.
First thing that comes to mind, you can use aliases.
To keep it simple, lets use 3 examples instead of 10: harlequin (this project), pgcli (https://www.pgcli.com/) and httpx (https://www.python-httpx.org/)
Setup a main home for all your venvs:
cd ~
mkdir venvs
Go into this dir, and create your venvs and install the packages cd venvs
python -m venv harlequin
~/venvs/harlequin/bin/pip install harlequin
Now this binary is available at $ ~/venvs/harlequin/bin/harlequin
Repeat for the rest cd ~/venvs
python -m venv pgcli
~/venvs/pgcli/bin/pip install pgcli
cd ~/venvs
python -m venv httpx
~/venvs/pgcli/bin/pip install httpx
Wash, rinse, repeat. Now you have all these binaries available and can alias them alias harlequin="~/venvs/harlequin/bin/harlequin"
alias pgcli="~/venvs/pgcli/bin/pgcli"
alias httpx="~/venvs/httpx/bin/httpx"
This is a pain in the ass though and usually simple CLI tools like this do not collide with each other. So that is why I say install globally, or install into your "global junk drawer" virtualenv.Meanwhile, for actual projects that you are developing on, those would have their own isolated venv.
I have a junk drawer venv where I install tools like this. If something goes wrong, it is as simple as rm -rf the venv and make a new one. And then I have isolated ones for each of the actual systems I maintain. Again, I use pyenv for this to make it a little easier to manage in conjunction with their specific python versions such that I do not ever interact with my distribution's Python. This is cross platform so it works across mac, linux etc. Very easy workflow, isolated, safe, can get blown away and recreated in a heartbeat.
Like, you're LITERALLY making a FANTASTIC argument for Nix usage in the Python ecosystem, here. In fact I'm going to bookmark this conversation now because of how ridiculously complicated your answer is compared to just using Nix.
Here's the Nix whitepaper. It's 14 pages or so. Read it on your next lunch break.
It's functionally identical to node_modules or Ruby Bundler or Perl local::lib.
It's so weird to me that people continue to hate Python for a problem that literally every programming language has had since shared libraries were invented.
Also, nobody active in the Python community will argue that there's 1 correct way to do packaging. That's a serious straw man.
Fortunately, none of those tools you mentioned other than venv are actually required to run Python applications, and there are in fact exactly 2 recommended ways to install Python applications:
1) Use your system's package manager
2) Use a venv, either manually (as shown in the sibling comment) or using the Pipx tool, which just creates venvs for you.
All of the other tools you mentioned (except for Pyenv) represent ~20 years of active development and iteration on how to manage projects and build packages for distribution, and end-users shouldn't even have to be aware of their existence. And Pyenv is just Rbenv but for Python.
As I've pointed out elsewhere, this is exactly the same situation as with literally every other programming language that doesn't generate standalone executables, and is even a problem with those that do, if they rely on dynamic linking. The special ire towards Python in this case is neither warranted nor valid.
Your pinned post on Twitter is predicated on double standards and lack of basic understanding of the tools you're criticizing. I'm not sure that's a good way to represent oneself.
If it means that I get to see less Python, then I guess... Mission accomplished? LOL
My first, second and tenth experiences with Python were all negative. Everything from trying to exit the REPL the first time I used it and getting chastised for doing it wrong (which meant that it knew what I was trying to do, and instead of just doing that, decided to be a little snit about it, which is about the jerkiest attitude a tool can take), to the Python 2.x->3.x transition pains, to the significant whitespace, to every project stepping on the dependencies of every other project (you might argue that "I just didn't use venv right" but I did follow every package's installation instructions!), to... Well, just read this, he did a good summary: https://medium.com/nerd-for-tech/python-is-a-bad-programming...
There's literally nothing I like about it. It just seems like a poorly written, older Ruby with a lot of baggage and a minefield of gotchas (except that I like Ruby... or did, before Elixir). Ruby should have absolutely fucking had Python's current market share, and I am 1000% convinced that if it did, everyone would be happier. Whenever Elixir tries to eat Python's lunch (like in the ML space which it's doing now), a part of me is as glad as a puppy. The language, despite its ubiquity, absolutely sucks in my mind, is not only a terrible introduction to programming for newbs but also likely an annoying language to work in, and the people who don't see it HAVE to be blind. That is the only conclusion I can come to, and I'm entitled to my opinion, strong as it is.
Next you're going to say that it is unprofessional to have such strong feelings about code and languages. To that I say: So what, dude. I care. Caring means having very positive feelings about the design decisions behind languages, or very negative feelings about the same. (I don't like Go, either. Python's basically a step above PHP on the "tech I want to minimize my time to zero with" department.)
Although your tool looks more like Virtualenvwrapper.
It's good to have other options of course (and yours looks nice), but it's also good to at least make a case for improvement over what's already out there.
I have, actually (though not Pipsi). Some others have commented on that and lead me to them.
Part of the "fun" of it was just doing it, I guess, and knowing exactly what went into it. Just a few lines of Bash, really.
If it's not hard then why is it not being done by default?
I am not a Python dev and I don't want to be, but when I have to install something via `pip` there's always pain.
"You could learn it", yeah yeah, Python is so special everyone has to learn it even if they don't ever work with it. How about learning from something modern like Go and Rust? `go install github.com/user/tool@latest`, boom, done. `cargo install the_tool`, boom, done.
Or ... is this a movement against application bloat, which has become all to common these days?
EDIT: why the downvotes?
But other than that, no, people don't love being in a terminal. It just happens that all open source portable toolkits like qt and gtk do not work via ssh and are in general total abominations for developers and users. The vt100 standard is 45 years old and turned out to be the lowest common denominator to write GUIs for better or worse.
Are you implying that people only write code inside of a terminal ssh session?
> A grid with monospace characters is perfectly suitable for that.
I've personally never seen a good experience of display forwarding over ssh.
Of course, most people and tools get around this by tunneling a connection to the database over ssh and running the GUI locally.
I would say the web browser is the lowest common denominator
Like have Helix or Nvim open, but you want to quickly check your database queries code changes. You can just open a new tab run the this sql IDE, fire off some queries check the result if it matches go back to coding new features. If not change the code, check queries again etc etc.
I switched to emacs during the pandemic because of Zoom and Slack (along with my horrible browser habits). VSCode is pretty reasonable on resources compared to many Electron apps, and I slightly prefer it to emacs in terms of overall experience. But emacs is also good, and there were just too many Zoom calls where my computer ran out of memory, with VSCode having a glaringly high footprint. I think at the time its terminal emulator was either excessively inefficient, or it had a specific resource leak. So maybe things have gotten better, but I've stuck with emacs regardless.
These days I can let a few dozen unread tabs in Firefox fill in the extra ~1GB of memory VSCode was formerly occupying :)
vim ncdu nnn mqqtui htop gitui tmux lnav lazydocker ctop
Also seems to be a bit of hubris to claim a SQL IDE "works with your database" when SQL Server and Oracle, two of the database products with the largest market share, are not supported (yet?)
I’m pretty pumped to try this seeing BigQuery is supported.
Agree on the first point, but is 'market share' - a metric comparing commercial sales revenue which by definition excludes open source software, really that relevant when critiquing a tagline for a FOSS tool implicitly targeting FOSS databases at a time when FOSS dominates?
If this wasn't an MIT licensed project without a single hint of a commercial offering I'd get it, but come on.
- stumble across the tool
- read that it "works with their database"
- hopefully read the actual list of supported databases
- become disappointed to learn that it does not in fact work with their database
In the case that hopes are not met, the user actually downloads the application and discovers the hard way that it does not work with their database.
So market share does seem relevant to the frequency of disappointment though. Is it going to be a frequent enough occurrence to be worth spending any time doing anything about? I don't know.
Alright so first off, for 90% of SQL Server's existence, Microsoft was openly open-source-hostile, so give me a fucking break with this. Microsoft used to be way worse than even Apple about keeping everything in their ecosystem- at least Apple is built on BSD underpinnings and was therefore also always compatible with POSIX stuff.
Oracle... last I checked on that monstrosity it had about 20 different clients all written at different times for different use cases over the long course of Oracle's history. So, which of these 20 different clients should one write a TUI for?
No we won't :)
Fwiw Microsoft SQL server is available as a Linux Docker build:
https://learn.microsoft.com/en-us/sql/linux/quickstart-insta...
Oracle is available as a container and vm:
https://container-registry.oracle.com/ords/f?p=113:4:1173021...