Hacker News Reader

> But it's just so damned useful to be able to reach into a library and get functionality the library's writer didn't think I'd need, that I'll personally never consider this to be bad.

I was ready to accept this as a matter of opinion, but then with this :

> Metaprogramming is the source of exponentially more technical debt (and bugs) than is reasonable, frankly.

Once you've jumped the shark and play with introspection and internals, are you really going to make the point that metaprogramming (or almost anything else, for that matter) should be banned ?

> are you really going to make the point that metaprogramming should be banned ?

As a fan of the concept of the "Catfish" developer (even though I'm a "corporate drone"), absolutely.

Or, if not banned, kept to an absolute minimum. It's like the preprocessor in C. You can do anything and everything in it... but should you?

You can be exceedingly clever with metaprogramming, but clever code is exceedingly hard to read and maintain. Give me boring, explicit code any day of the week.

Metaprogramming goes way beyond what the C preprocessor does.

Metaprogramming also cannot necessarily be "hard to read and mantain". Take a look at macros in Common Lisp. They are almost the same as a normal function. Yet they do metaprogramming.

> Give me boring, explicit code any day of the week.

With macro metaprogramming you can eliminate boilerplate code / unnecessary copy-paste repetition of code, and this does improve maintainability of the code.

At the same time, replacing 10 repetitions of the same big boilerplate code (similar code, written with slight variations each of the 10 times) with 10 very simple (one line each) calls to a macro, improves readability: because you will easily read what is the difference between those 10 calls each.

Macro metaprogramming can (and in practice often does) break ctrl+f search. If I want to find the definition of PREFIX_varname, I'd expect to grep for "PREFIX_varname", but often, the correct solution is to search solely for varname, because PREFIX_varname is defined by a call to MACRO(varname).

Maybe don't use such a primitive editor integration?

I'm not trying to dunk on your editor, but people have had this problem solved since the early 90s. If your programmer's editor doesn't have some affordances for your language, then it's not a programmer's editor at all, it's a text editor.

I'm curious: what editor can do this? Consider that the codebase I work on is large enough that it cannot fit in a project file or ctags or similar. (Which is to say that VS, Clion, and others don't work, nor will vim (for this specific feature))

For which language? In the case of systems like Clojure, if you're introducing new toplevel bindings you're not going to get completion without direct interactive evaluation (naturally). But everything short of that should be well-formed.

In many cases with common macros I'd add decls for Cursive Clojure + Idea to let me autocomplete introduced bindings. For example, I helped my editor understand unique structure, e.g.,

    (defauthedendpoint get-frobnaz [request user] 
        ...re<tab>quest)

C/C++. In the general case, I don't believe its possible to resolve macros without linking.

Actually, I'm not sure. If we're referring to the C pre-processor here, can't we always just run it through before symbolizing?

I guess that without the right environment ifdef stuff falls out, but I dunno what to tell you there other than making builds contingent on the environment is a sketchy practice.

So, say I have this:

    #include a.h
    #include b.h
    #include c.h
    MACRO(varname)

welp.

Since the include chain has to terminate, can't we just run the pre-processor over every file before we index it?

Seems, um, like a pretty reasonable idea to me if you care about C-style macros (I sure don't.)

It may be sketchy, but you may not have a choice. If you have a Qt application (written in C++) which needs to use a platform-specific API to do some magic stuff with a window, there's a method on the window to get the handle that the platform's GUI library offers. Of course that's going to be dependent on the environment because each environment has an entirely different GUI library (e.g. Cocoa only exists on macOS, not on Windows or Linux or Android).

I can't parse this in a way that it makes sense; I suspect you're mixing up some terminology. I'm guessing you mean that you need access to all transitively included headers to know a macro's definition. That's certainly true, but it's not linking, which involves resolving symbols in the object files that are the output of the compiler. The C preprocessor logically runs before the compiler; its output forms the compiler's input. You can run it without running the compiler at all.

Ack, you're correct.

On the other hand, if my editor has to incorporate a full-blown parser for your program's (meta)language in order for me to find where something's defined, then maybe your program needs refactored more than my editor needs replaced. Or perhaps better documentation on your program's part and/or better education/familiarization on my part is warranted.

Most modern editors are able to horizontally scroll and thus handle long lines in source code. That doesn't make a 500-character-long line of code acceptable by any stretch.

> On the other hand, if my editor has to incorporate a full-blown parser for your program's (meta)language in order for me to find where something's defined, then maybe your program needs refactored more than my editor needs replaced.

Either you want your editor aware of your language's semantics if you don't. I rather do, because it saves a ton of time and basic integrations are usually quite straightforward in 2017.

> Or perhaps better documentation on your program's part and/or better education/familiarization on my part is warranted.

Barring my opinions on this, I don't get why this would exclude editor semantics.

> Most modern editors are able to horizontally scroll and thus handle long lines in source code. That doesn't make a 500-character-long line of code acceptable by any stretch.

Okay, but this is a false equivalence. You're telling me not to use metaprogramming tools because they make it hard for a specific editor feature to be used in conjunction with them in the most difficult case for any system (introducing new run-time-dependent lexical bindings into a scoped block and handling that at compile time).

But unlike code visibility, in most good solutions exist to work with these systems (within reason).

You are speaking about C macros, which have almost nothing in common to Lisp's macros.

There is no such problem with Common Lisp macros.

There could be such a problem with Lisp macros, if the programmers are complete idiots.

C programmers gluing together symbols from pieces so that MACRO(PREFIX) expands to some PREFIX_var or whatever are often not complete idiots; they are desperately doing whatever they can to simplify what they are doing in the best way that is supported by the portable language.

Lisp macros go far beyond what is possible in Python/Ruby, and oddly enough are less of a problem because of that. In Python and Ruby, the inability to change the underlying syntax makes alot of developers go to absurd lengths to create new syntax around the existing syntax. And that's what makes Python/Ruby metaprogramming really painful to understand.

That said, my (admittedly few) days spent debugging bad macros were pretty damned bad. Having to inspect the results of the macro expansion and figure out what went wrong is not my idea of a good time. And that doesn't even account for reader macros.

If a macro is correctly expanding, but you have to go into the expansion to find out what is going wrong when the macro is misused, that means the macro lacks robustness; it is not checking arguments, or not sufficiently or in the right way for the cases you're running into. Bad arguments make their way into the expansion and then trigger errors which are conveyed in terms of the expansion, not necessarily making sense to the macro user.

It takes extra time and effort to make macros that are suitable for broader use (people other than the macro writer).

Agreed on all points.

But ultimately, bugs in software happen, and bugs in macros can be a pain in the butt to debug. The extra steps to go in and expand the macro, and then translate the fix in that expansion into a fix for the macro itself gets complicated, fast.

The job of the macro is to convert a form into an expansion. If it's doing the job wrong, I don't see how you can avoid looking at the output. With respect to what is that "extra".

It's no different from any other "input -> process -> output" computing situation where the output is wrong, the right output is obvious, and you have work back into the process to make that output come out.

You shouldn't be fixing macro output. If the macro is wrong, the first step is to find the simplest macro invocation which reproduces the problem. Use a dummy argument for anything irrelevant, and use an atom for any argument that allows one. For any argument position which accepts a list of items, see if the problem reproduces with an empty list, or a list of just one. If you can see the problem in (mymac (a b) c d), you have it made. Hey look, b is being wrapped in an extra list, and d should be quoted."

> You can be exceedingly clever with metaprogramming, but clever code is exceedingly hard to read and maintain. Give me boring, explicit code any day of the week.

Boring, explicit code which depends directly on the unspecified, unsupported, implementation details of the libraries you're using ... ?

I mean that's the point of tooling. Metaprogramming removes any advantage and IDE can give you to make sense of unknown code.

This is strictly false though. IDE integration can give insight into metaprogramming in many languages.

Python, the language in question, does not easily offer this support, since all of the metaprogramming occurs at runtime. You have to run a debugger while creating the object to see what is actually happening.

Lisps, C Preprocessor (within certain limits), and so forth are a bit different, since you can expand the macros without having to run the code itself.

Does that indicate a strength of Python or a weakness? A complete value judgement there. In either case, metaprogramming in Python (and many other languages, even those with support for metaprogramming resolution in their IDEs) is still simply harder to understand and debug.

I feel no real problem shaming the entire design of Python as a long bet on failed premises. But other languages with similar properties can and do report their metaprogramming results to language servers.

Agreed completely—this is the difference between metaprogramming and "black magic", which is familiar to any ruby programmer.

Yes, that explains why the language is failing.

Even that IDE known as Vim + Ctags will jump to a "defmacro" definition if your cursor is on a macro call and you type Ctrl-] to chase a tag.

This only works for metamagic identifiable macros. What if you refer to an identifier that doesn't exist anywhere in the code? What about ruby's method_missing? It's impossible to provide tooling for those cases without running the compilation or evaluating.

Well, I also commit the faux pas of recommending limiting external dependencies to the standard library in Python whenever possible, resulting in many fewer dependency issues than so many of my colleagues.

So, while occasionally under specified, they are quite well supported and stable.

I do miss out on the fun of sorting out the requests dependency chain with every release of my code, but I'm mostly OK with that.

It's weird how people neg on some abstractions for Python but then brag about the absolutely intense degree of abstraction something like Tensorflow brings to the table.

It's almost like the qualification is that the reader can't even imagine writing the abstraction and therefore they can ignore it.

There's a huge difference between abstraction you have to write and debug yourself, and abstraction that you can use and debug through documentation and StackOverflow.

Third-party resources are where it's at for taking advantage of things that require skills you don't have. I haven't written enough frameworks and abstract libraries to expertly debug, test, and write my own.

> There's a huge difference between abstraction you have to write and debug yourself, and abstraction that you can use and debug through documentation and StackOverflow.

Indeed. You're more likely to be able to use and debug it if you wrote it yourself rather than relying on the weirdom of the crowd from SO.

> I haven't written enough frameworks and abstract libraries to expertly debug, test, and write my own.

I suspect you'd do just fine if you tried, assuming you don't try and displace django.

There is a material difference between syntactic abstraction and functional abstraction. It's analogous to the difference between someone learning to program a computer and someone learning to use a computer. The second is far easier to do and requires far less intellectual overhead.

By the same token, if someone implements TensorFlow in a different language, there is little room for complaint assuming the same functional abstraction (and minimization of intellectual overhead) is present in that language's implementation.

"Reaching into a library" means depending on something not meant to be public, with the understanding that it is undocumented and subject to change without notice. Giving a reliable but soft warning (through quasi-encapsulation naming conventions and explicit module exports) and assuming developer maturity is the typical Python approach to double-edged features.

Hacking something with private library parts doesn't involve any more "introspection" and "internals" than reading the library's code and finding useful loot to steal; metaprogramming is a completely different and orthogonal way to compromise code maintainability.

Once you've jumped the shark and play with introspection and internals, are you really going to make the point that metaprogramming should be banned ?

"Banned", no. Just that's not simply this big trove of goodies that it's often cracked up to be. And in particular, it isn't strictly speaking necessary or essential for a language to be successful -- or even among the top 5 considerations that make a language successful or not.

"WRT private class members, I can understand why this might be frustrating for library writers. But it's just so damned useful to be able to reach into a library and get functionality the library's writer didn't think I'd need, that I'll personally never consider this to be bad."

I'll second that. I'll share my own experience to give some perspective. Older versions of the JDK had an LDAP library that for some reason I can't understand (actually pretty sure it was a mistake because it was changed in subsequent version) specified a File as the type instead of a Stream for its configuration file. There was no way for me to modify the configuration during run time without restarting the JVM. This wasn't acceptable to the software we were writing because it supported multi-tenancy.

What did I end up doing? I used introspection to grab the "private" configuration table (Vector or Hashmap?) and wroten an API around it to allow modification on the fly.

Lesson here is that "private" things aren't really so private and Python style discretionary "private" variables would have required a lot less work.

Is it possible that encapsulation just doesn't work, in general? It really boils down to a library author saying "trust me, this'll work great!" Sometimes it does, and sometimes it doesn't. In that light, Python's convention that a leading underscore means "probably better avoided" seems pretty reasonable. This is especially true for Python, where the type system gives you no guarantees about anything.

At my previous company, we ended up writing Python code that looked an awful lot like Scala or Java:

    class PricePredictionCalculator():
        def __init__(self, price_prediction_calculator_settings: 'PricePredictionCalculatorSettings'):
            self.x = price_prediction_calculator_settings.x
            self.y = price_prediction_calculator_settings.y

        def calculate_something(self) -> int:
            return self.x + self.y

    class PricePredictionCalculatorSettings():
        def __init__(self, x: int, y: int):
            self.x = x
            self.y = y

It was actually kind of nice. It allowed rapid prototyping without tons of boilerplate, and then tacking on the boilerplate at the end. Mind you, I was a data scientist, not a full-time engineer. There was also a lot of room for stylistic differences and varying opinions.

Regarding your variable naming: why `price_prediction_calculator_settings' and not, say `settings'? Same with the type; why not:

    class PricePredictor:
        class Settings:
           def __init__(self, x: int, y: int):
               self.x = x
               self.y = y
        
        def __init__(self, settings: Settings):
               self.x = settings.x
               ...

That battle was lost before I even had a chance to fight it.

I'm curious why about why you don't like that name?

I'm asking because I'm usually torn between short names that assume the reader can understand them and long overly descriptive names that (in theory) require less implicit understanding.

Is the name price_prediction_calculator_settings too long? If we call it settings I can see an issue come up if we need to pass in another type of settings object calculator_format_settings or something like that.

This is one of those areas where you're actively being hurt by the need to name something or give it a type. Wouldn't passing in a simple dictionary for the settings be simpler? Something called PricePredictionCalculatorSettings probably isn't being reused elsewhere.

The whole idea was to get away from passing dictionaries around.

There's no IDE support for "plain" dictionaries, they resulted in code duplication and broken encapsulation since you can't add any logic to a dictionary, and they put unnecessary cognitive load on the developer by forcing them to keep track of which fields go in which dictionary.

I saw it as a form of design-by-contract and self-documenting code. The app I worked on had a ton of these mappings being passed around between different subsystems and it was becoming nightmarish to deal with. With this style, if you were on Team A and needed to plug your work into Team B's FooBar class, all you had to do was look at the FooBarInputData class to see exactly what the FooBar class needs.

Interesting point. If you were using Clojure I'd suggest checking out Schema and spec. They allow you to define the expected fields of a map/dictionary, without requiring a full type, which is handy for well-defined data objects without attached behavior.

Overly-long names are an indication that the design could use some refactoring. There's a tradeoff between variable name length and the amount of readable logic you can fit into a line.

The name ``settings`` is too generic. There's probably a better way, but to demonstrate that we'd need a real example and not hypothetical.

Long names require lots of keystrokes and more reading effort. Also you have to split your lines earlier. If I have long names, I won't remember what I called things, at which point I'm leaning on autocompletion. Hitting tab every few keystrokes really takes me out of my flow.

The other general issue is why have the redundancy? There's only one settings object here. If you need to pass something different in later, refactor! Remember the principle of You Ain't Gonna Need It.

At that point why not just write Scala though?

On the plus side of writing it in Python, no one can type "import scalaz" and start writing brittle backwards haskell-ish in your project, causing massive performance regressions they will defend to the death on grounds of leaky abstractions.

Even Scalaz-heavy code runs a lot faster than equivalent Python, so I don't think that argument makes any sense.

It's a bit of a joke, but certainly avoiding the majority of the scalaz community is a big plus for anyone using Python. You'll never have to endure another Tony argument, for starters.

Yeah, that is a big black mark on Scala. Just an unfortunate failure of leadership - I'm not convinced any other language community would handle an individual like that any better, but the Scala community did not acquit ourselves well to say the least. I dare to hope that with the rise of Cats that kind of thing will cease to be tolerated and the language community can become a pleasant place, especially for newcomers.

I mean, Tony tries to edge into the Haskell community and gets rebuffed. He simply can't neg the people there, they're too busy actually producing novel results rather than talking about them.

Some Haskellers love Tony. I'm one of them.

To be fair to him, Tony is much more well behaved in the Haskell community because he doesn't seem to feel the license to run roughshod over everyone there.

Anyone who says his behavior hasn't been troublesome in #scala at times over the last 5 years isn't paying attention.

I still remember my first run in with him where he demanded I take a "test" to let him "grade" me. In the middle of a larger conversation about a sbt bug.

Yeah, I meant to say any language community at a similar maturity level - he was there in the early days of Scala and wrote a foundational library that continues to be extremely widely used, which creates quite a different dynamic.

I think "Yeah, that's good, but why not use X?" is Python's biggest problem right now.

At this point, The Zen of Python has been absorbed into many late-gen programming languages; it's easy to see a heavy Pythonic influence on Swift and Go, for example, and some languages, like Nim, even draw the parallel as a promotional thing.

Sadly, the technical underpinnings of the Python runtime itself have not kept up, and it leaves people asking why they shouldn't just enjoy the same advantages they'd get with Python through a newer language with a modern implementation, providing better performance and package/dependency resolution. As much as it pains me to say it, even cutting-edge ECMAScript code can be made to look pretty Pythonic these days.

Python will always hold a special place in my heart, but I'm not sure that a Python implementation is the obvious choice for a dynamic application anymore.

> WRT private class members, I can understand why this might be frustrating for library writers. But it's just so damned useful to be able to reach into a library and get functionality the library's writer didn't think I'd need, that I'll personally never consider this to be bad.

I can only really think of one instance where I've used this, and it was to work around an outright bug in the library (which was unmaintained, and I was using out of a certain amount of desperation).

FWIW, most of the complaints around private in my post would actually be solved just by having a separate namespace for "private," even if there was a back door. I'd be pretty willing to forgive that.

> Multi-line lambdas would certainly be nice, but most of it can be handled with a scope-level function.

In the absence of the with statement can you really imagine yourself writing this all the time:

    def _with_body(f):
        print(f)
        print("I am a teapot")

    with(open("foo.txt"), _with_body)

?

Strikes me as pretty ugly, despite not actually adding much code.

> As for syntax, no need to use Ruby's syntax, just incorporate parenthesis (a well established continuation construct within Python):

Yeah, that's a bit nicer.

> As for the example error - there is a typo. Of course it's going to throw a traceback. The error message is even more explicit than I expected, frankly. It pointed out the missing function name, which would make the typo rather easy to find.

Yeah, that was kinda my point; I was comparing this to what happens if you try to "take duck typing to heart", where you don't get the error until you try to call request. The earlier you find out about mistakes the better.

> can you really imagine yourself writing this all the time

Honestly, no. But not because of any measures of beauty, but because I just don't do callbacks all that often.

    try:
        f = open("foo.txt")
        print(f)
        print("I am a teapot")

    finally:
        try: f.close()
        except: pass

is what I would be more likely to write (and did write, before context managers were a thing), if the "with" statement didn't exist. It's ugly as sin, but I can put it in front of anybody and they will understand what's going on. I can also tell at a glance that it is correct.

Re types - actually, mypy is getting very good, i highly recommend checking it out. It smells a bit like ML/Rust except there's no pattern matching.

There is no excuse for missing multi-line lambdas other than negging on programmers.

Really, is it that hard to come up with a name for your function?

I don't even understand why your question is relevant.

If you want a multi line lambda:

    def print_each_twice(xs):
        map(lambda x: print(x) # whoops, impossible

You can define a local function:

    def print_each_twice(xs):
        def print_twice(x):
            print(x)
            print(x)
        map(print_twice, xs)

The former is more intuitive to write. The latter reduces nesting, makes you name the function, and improves readability IMO. It's currently easy to scan blocks of Python and see each indentation level starting with one of a few keywords (class, def, for, with). Adding multi line lambdas would make the ultra simple structure harder to quickly parse.

> If you want a multi line lambda:

I don't want this though. This doesn't do what multi-line lambdas do. It can never do what lambdas do. It's not a lambda function if it's named and fully pre-computed.

> Adding multi line lambdas would make the ultra simple structure harder to quickly parse.

This is just about the least compelling argument I've ever heard. "The parser writers are not going to have fun."

I meant to parse mentally. Control flow in Python is very easy to mentally parse at a glance. Powerful lambdas get rid of that.

It's an extremely contrived example, but from the code I posted, it should just be:

    for x in xs:
       print(x)
       print(x)

Much easier to read because there's only one way to write it. Hence GVR shunning map/reduce/filter.

GVR is wrong. It's okay to be wrong. But the last 5 years of software development and the remarkable growth of Javascript as a platform accessible to everyone make a compelling case that GVR was wrong.

No it doesn't. It makes the case that the browser is the most pervasive and flexible application runtime, which we already knew. Server-side JavaScript was never a thing until that was solidified, thus providing a large developer base.

Why is this a meaningful distinction? Who cares if it's server or client side in this discussion? The Python community has said passing anonymous closures is too complex, the majority of the industry disagrees and has pushed forward.

Lambdas are everywhere BUT Python now.

Javascript is even pushing forward with more sophisticated constructs to help them build custom monadic constructs, and shipping a super useful monadic construct in syntax. C# has been doing that for years. Java will soon as well.

The rest of the industry has moved on from this conceit and you can't really argue that it's made environments like Javascript overwhelming to newcomers.

If Python 3 could just get over this conceit, its maintainers could start solving major Python problems really quickly. E.g., Python's janky iteration primitives could be replaced in an afternoon or three with equivalent but cleaner and clearer options.

> Why is this a meaningful distinction? Who cares if it's server or client side in this discussion?

It doesn't really matter, except that the browser's dominance as client-side platform is the cause of JavaScript's current ascendancy, rather than programming language design decisions of any sort. This is a refutation of the claim that you made saying "the last 5 years of software development and the remarkable growth of Javascript" make a compelling case that van Rossum is wrong about multiline anonymous functions in Python. Classic post hoc ergo propter hoc fallacy.

If you're going to stick by the argument that somehow the support of anonymous functions is responsible for JavaScript's recent successes with respect to Python in terms of adoption and pace of development, then you'll have to explain why Lisp didn't take the world by storm when they were introduced nearly sixty years ago.

Or, of course, admit that whether or not a particular programming language conforms to your personal preferences has relatively little bearing on the success of that programming language in the world at large.

Maybe I misunderstand what you mean with "fully pre-computed", but local functions in Python close over local scope at runtime:

    >>> def splatter(t):
    ...     def f(): print(t)
    ...     return f
    ... 
    >>> beep=splatter("beep")
    >>> boop=splatter("boop")
    >>> beep()
    beep
    >>> boop()
    boop

That's the problem with Python, everything is dynamic.

I meant structural precomputation. But yeah, I forget you can discard names. You're right: through sufficient use of discarded intermediate bindings you can replicate multi-line lambdas.

Still, the existence of this sorta demands we ask, "So why can't we use it then?"

I'm not steeped in functional programming, but isn't anonymity the least interesting part compared to being first class and forming closures?

I don't really see it as an FP thing at all, because the same argument can be made with respect to variable names. The issue is that without serious anonymous functions, you're forced to come up with more names, increasing cognitive load. An anonymous fn clearly communicates "I'm not going to be used elsewhere".

So call it something generic related to usage: keyfunc, repeat, parse, callback, etc.

Why increase the complexity and density of the code while complaining that the alternative would be too complex though?

If I can simulate anonymous functions with locally scoped functions that then behave as lambdas then what hill is Python dying on here? The "we don't want to submit a patch to make our parser do this?" Hill?

It's definitely possible to do. So...

The small increase in cognitive load when writing the code pays off when reading it. My editor can tell me where a function is used. It can't tell me what a function does.

And with anonymous functions, your traceback often hasn't a clue where the function was defined.

That's untrue. That's usually the least interesting part of the stack trace, but the easiest to compute.

But your complaint isn't without merit here, a profusion of unique nameless verbs can get overbearing sometimes. It's often better than the alternatives (e.g., dependency injection vs lambdas in an environment) but Haskell, ocaml and lisps all seek to have a small but very robust set of combiner and base primitives to help make this kind of code avoid a complexity explosion similar to OO ontology explosions.

Huh? What do you mean? That multi-line lambda is not standard Python is it?

No, it's not included. I meant: "It would be nice; here's my suggestion for syntax"