The Dynamic Def – abusing Ruby's def statement
weblog.jamisbuck.org
weblog.jamisbuck.org
http://jamesgolick.com/2013/4/14/mris-method-caches.html
The late, great James Golick had a patch to add one once, but it never got merged upstream.
If you care about performance even the tiniest bit at all whatsoever, please don't use the techniques discussed in the OP in production code or in your gems. It may make your memos 30% faster on a microbenchmark... while causing the rest of your program to run considerably slower.
It remains an open issue: https://bugs.ruby-lang.org/issues/9262
This stuff feels really dangerous and unnecessary. I spend a lot of time on code reviews pointing out bad features of Ruby (and Rails) that we shouldn't be using because they break application flow and make it significantly harder to reason about the code for the small benefit of decreasing a few lines. But it's certainly fun to talk about :)
Is this just a complaint about the ability to overload operators in general?
Instead of overloading, I've always wanted to define new operators e.g. <DotProduct> used as 'int x = v1 <DotProduct> v2;'
If you're going to do something with operators, at least let me make descriptive ones.
That's a great way to put it.
Instead, maybe a method MatchAttr1And2(v1, v2) would certainly tell a subsequent reader a little more about what's going on.
This is, fundamentally, a disagreement about the value of encapsulation. With an opaque, encapsulated type, '==' should mean whatever makes the most sense in the context of that type. For a pointer that might be "equality" means "same memory address", whereas for a vector that might mean "equal components". As a user, "equality" should match an intuitive understanding of what it means for two things of this type to "be the same". It's an art form. Like many things in programming, doing it well requires good taste.
Operator overloading is a powerful technique for preserving encapsulation. It's the polar opposite of:
> Instead, maybe a method MatchAttr1And2(v1, v2) would certainly tell a subsequent reader a little more about what's going on.
This leaks implementation details like a sieve. It's a great recipe for encouraging dependency on a particular implementation detail across module boundaries, and rolling yourself a great heaping ball of mud.
Operators are unique in the language. They hold a special place. They deliberately are written to imply something we already understand. No fair lumping them in with every other attribute or method of an encapsulated type.
Intuition is a very, very poor thing to depend upon in a programming language. I disagree heartily that it should be the solution to disambiguating any operation.
Making the name more opaque won't save you at all. You're making what should be a local detail -- how your type implements equality -- into something that only works when it is global knowledge.
Consequently, your type is brittle and incomposable with types that aren't infected with this knowledge:
[instance11, instance2, instance3].sort
Every type you build in this manner will find knowledge about itself diffusing throughout your application, like children peeing in a pool. Composition will be limited, inflexible, and require manual insertion of type-specific knowledge, because you have failed to encapsulate knowledge about equality.
Everything needs to know about everything else, and in the end you've built a tightly coupled ball of mud.
I would argue that a CORE value of Ruby is that everything is an object, and objects communication by message passing. Treating an operator as anything other than a message between objects is fundamentally wrong.
If you expect operators to do anything other than call the appropriate message on an object, you're misunderstanding the syntax.
Ruby only has messages. Some of those messages just happen to have punctuation for names.
That 5-component object compared by ==? How does it work? Sit down, read the code and find out. The answer depends on what exactly the object represents and what makes sense in the domain model.
Its too simple to do victim-blaming here. You don't understand my code? Well, just read it all so you know how clever I was.
If you want to write code that can be easily assimilated, which most readers would think they understood from its source and not by reading it at some meta-level, then you have to code with one hand behind your back.
I generally disagree with people who dislike a language feature only because of its abuse potential. Good programmers should not have to suffer for the sake of damage controlling bad programmers.
Edit: I do think it's stupid to make the identity-equals operator overloadable. Identity-equals and value-equals are separate concepts. In C++ this isn't an issue because == is the value-equals operator.
In Ruby, == is value-equals at well (identity-equality being such a rarely needed concept in Ruby that it wouldn't make sense to privilege it with its own operator).
Identity equality in Ruby is provided by:
a.object_id == b.object_idAnother trap is redefining #== without also looking at #eql?, which means Hash doesn't behave like you expect. It's just another bit of mental trivia you've got to Just Know...
Lisp has 'eq', which is for object identity. Lisp has 'equal' which checks for structural equality.
The default `==` from Object compare identity, so unless you define it `Money.new(20, 'USD') != Money.new(20, 'USD')`.
Don't try to change or complain about Ruby, you are likely more happy with languages like go.
That's not the issue at all. Abstractions are great. Redefining operators is not abstraction, it's practically obfuscation.
The fact that it's an operator or a method doesn't change a thing. For example in Java many classes redefine the `equals` method. It's default behavior is just like Ruby, comparing identity. It's not an operator but the effect is exactly the same. And IMO it's worse because now you have a leaky abstraction with types you need to compare with `==` and others with `.equals`.
It's just you who have this expectation of operators being not redefinable. When I read Ruby code, for me `==`, or `+` are just regular methods like any others with just a bit of syntactic sugar.
It also allows for greater polymorphism. Like the Money object from before. If I couldn't redefine `+`, then `[Money.new(20), Money.new(22)].sum` wouldn't work.
Or Money.new(20, 'USD') == Money.new(125, 'CNY') when ExchangeRateManager.getExchangeRate('CNY', 'USD') == 0.16?
My point is that when performing these comparisons, it may be useful to use a more descriptive function like:
boolean currenciesHaveSameWorth(Money m1, Money m2)
And then a reader of the calling code might not have to look into the implementation to understand what the function is doing, whereas you definitely would when using == because == now means "whatever it's overridden to mean"
Like someone else pointed out in this thread, designing APIs require consistency and good taste.
If I had to implement this API you code above would evaluate to `ArgumentError unknown currency "US Cents"`.
> Or Money.new(20, 'USD') == Money.new(125, 'CNY') when ExchangeRateManager.getExchangeRate('CNY', 'USD') == 0.16?
Again, me designing this API, it wouldn't be equal. Why? For the same reason `1 != "1"`, if you cast them, yes they are equal, but implicit casting (aka weak typing) is not idiomatic in Ruby, it's possible, but very rare.
> boolean currenciesHaveSameWorth(Money m1, Money m2)
At this point you might as well do `m1 == m2.convert_to(m1.currency)`, because "HaveSameWorth" might mean many different things too.
I personally hate that last style because it's obvious that the "HaveSameWorth" relation is intended to be symmetric, and by writing it like m1 == m2.convert(...) you're prefering one side over the other. It looks bad for me :).
Also, in case of real-life objects it makes sense to spell out what do you mean by 'equality' (or 'equivalency'), and leave the default implementation to represent the philosophical concepts of "the same" and "equivalent to".
For example BigDecimal. With operator overloading, I can easily compare a BigDecimal object to an integer or float, the same way I can already compare them to each other:
BigDecimal.new("10.0") == 10.0
10 == 10.0
d = BigDecimal.new("10.0")
!d.nil? && d.value_equal_to(10.0)
However, the language itself is great. The trick is to remember the usual advice that just because you can doesn't mean you should. Too many gems add "clever" metaprogramming (such as the def tricks in this article), when it isn't actually making the program simpler.
The beauty of ruby is that while it can act almost like a LISP for the times when you want powerful metaprogramming features, while also allowing simple shell or C style imperative code when that is more appropriate.
(Of course, some people fear that type of freedom... https://vimeo.com/17420638 )
But still, this is a really cool trick that I didn't know you could pull off with Ruby, so thanks for sharing!
Honestly, I wish there were more people doing posts about recreational programming topics. WhyTheLuckyStiff was one of the last great recreational Rubyists. I miss that kind of no-holds-barred exploration.
I hate this "dynamic programming[0] = hard to debug" meme. A truly hard to debug code is one that's nonlocal (you have to read 20 files to follow the execution flow (hello Scala trait abusers)) or displays random behaviour (e.g. threading, dependence on external resources). This one? Every "tricky" thing is contained in a block of 10-20 lines. Just isolate the endpoints and follow the execution until it does something you think it shouldn't.
In a way, the biggest enemy of a bug hunter is their own assumptions.
[0] - examples here aren't really metaprogramming, and even the latter isn't that hard to debug if you actually sit down and read the code.
In Ruby, no visual indication exists. Worse, things like attr_reader blend instance variables with local variables and method identifiers. Throw in a method_missing and inheritance, and you can easily lose weeks just tracking down where an identifier is even coming from. Throw in a gem or two, and all hope is lost.
It may not be depending on the context; consider:
(let ((foo 123)
(+ 'please-dont-do-that))
(print foo)
(print +))
But one thing Lisp does have, which impacts readability significantly IMO, is simple and consistent syntax. Contrast to some Ruby-like languages which let you skip braces when working with dictionaries, making you stop and wonder how the hell a given piece of code is going to be parsed by the interpreter. Or Scala, which has so much context-dependent meaning bound to non-letter characters that I finally start to understand why people were afraid of C++ operator overloading. Both examples are, in my opinion, cases of syntactic sugar leading to cancer of semicolon.
def some_method
what_is_this
end
> In Lisp-n (like Common Lisp) you can have a variable slot bound to a function value (i.e. lambda).
Yes, and Lisp-1 vs Lisp-n has been a hot debate for decades in both Lisp and Scheme communities. I'm not about to claim it's a completely solved problem there.
def some_method
what_is_this
endWe know it's not an instance variable -- that would be @what_is_this. It would have to be a local variable, but plainly there is no such variable local to this method. So we know it's a method. Let's go hunting (we could do this vis binding.pry, or byebug, or in IRB with an instance of whatever defined some_method):
method(:what_is_this) rescue false # if false, this is coming from method_missing
method(:what_is_this).source_location # there's your definition location, if it wasn't coming from method_missing
Then again, you could make similar shenanigans in Common Lisp with `symbol-macrolet', but it's obscure feature that pretty much by definition will be used only by people who know when to use it :).
Definitely agreed! It's rare that I've come across a set of code that is _so_ over-abstracted or dynamic that reading the code doesn't usually shed light on the issue I'm after, and usually it's a "dense" language (Scala, Lisps, etc.) that manages to achieve that.
It made some sense in the end, but it took me a lot of time to figure that one out - not because it was complicated, but didn't fit in my head. That the original author was uncooperative and didn't want to explain things too much didn't help either.
Personally, I see this as the entire point of the original Smalltalk-esque OOP paradigm: an object is a living, mutating black box that responds to messages, where an object's "type" is just "object": a thing with a protocol for sending and receiving arbitrary messages, not a thing with particular messages it is lexically known to respond to.
In other words, an object is like a remote server on the Internet. You can no more introspect an object by looking at the source of the classes it was originally constructed from, than you can introspect a remote web service by reading the source of the frameworks it was based on. The set of messages an object will respond to, and how it will respond to them, is part of the object's state, not part of its definition.
This "absolute encapsulation" forces the producer to publish an API if they want anyone to consume their object/service. This is great! If the API is machine-readable, and the object/service publishes it as a response to a message, an OOP runtime can even perform runtime introspection on the object/service, enabling consumers to configure themselves to speak another object's protocol, to reconfigure when that protocol changes, and even for two objects to negotiate a communications protocol among many options.
People have rediscovered the benefits of black-boxes with published, oft-machine-readable APIs in the last few years, calling the new version of OOP "microservices." It's the same idea: objects with no lexical type beyond "thing that responds to messages encoded in this format", being used as black-box interfaces into libraries which may be running locally or remotely.
This is also, effectively, the definition of a "process" in Erlang: something that will (asynchronously) receive messages if you send them, and might send you a response, or might not, with the interpretation of a given message depending entirely on the process's state. (People who say Erlang is a functional language are looking on the wrong level of abstraction. Processes are objects!)
---
To get back to Ruby, though: imagine an object which serves as a REST client, where the method_missing of that object translates the {method_name, ∗args} into a GET request to an API server, with the method name becoming the path and the arguments becoming query parameters. This is an idiomatic kind of Ruby object, because Ruby is actual-OOP rather than the "classes are types, right?" faux-OOP of static languages.
The messages this REST-client object responds to depend entirely on code running somewhere else that could be modified at any time. There is nothing static analysis tools could do to figure out what this object will or won't respond to. The only way to introspect its operation at all is at runtime.
And yet, I would argue that this implementation of such an object is the best, most 1:1 translation of the concept of "REST client" into a programming language. Errors are propagated from the remote server, through HTTP, into local dynamic-dispatch errors, through the defaults of Ruby's runtime. You're not having to go against the grain, writing a "get" method and then making up all sorts of custom exceptions it can raise. You just make a local object, that stands in for a remote object, and then you interact with it as an object.
People thinking those two are mutually exclusive are not getting the difference between the map and the territory :).
I worked a bit with Erlang professionally. Erlang is both functional and object-oriented, if you think about Smalltalk-style OOP and not Java-style OOP (the constant confusion between the concepts of those two is incidentally why I think that most things written about OOP are bullshit - they focus on wrong details; I've seen even university courses confusing the shit out of students by calling C++ methods 'message passing' and objects as equivalent to 'actors').
RE the problem of black boxes - black boxes are cool, what's not cool is if the interface you use to talk to black boxes is in itself confusing. In the example GP posted, it's not clear on first and second thought what exactly does the middle line mean - if it is a value, evaluates to a value, evaluates to a value with side effects, possibly taking your control flow for a sightseeing trip around the Moon, etc.
Also, in real world, we have to have some idea of what the black box is really doing. The only value that comes from assuming something is like "a remote web service" is knowing that it probably sucks, it's totally unreliable, if it responds at all it's after time noticeable to end user of your program, and you have to plan for it disappearing at any moment because the founder takes exit money or forgets to renew a domain. Yes, you could program treating everything like a web startup, but few simple assumptions like "this is inside my program so it responds fast and lives as long as the rest of the program does" can help tremendously improve the speed of coding and the speed of the program itself.
I think the distinction between regular Smalltalk OOP (designed before distributed programming was a thing) and what people actually mean when they call a language "actor-modelled" is that, in a language like Erlang, you get to lean on the language itself (or in Erlang's case, the OTP framework) to be pessimistic about other objects' behaviors for you.
One thing that I think is missing from today's OOP language landscape, though, is a concept of protocol parameter adjustment for long-lived peers, ala TCP window scaling. While I wouldn't expect this of anyone's one-off RPC protocol, the frameworks like OTP, specifically made to cleverly handle OOP RPC stuff, should be capable of multiple levels of "formality" in how objects speak to one-another, where an object that e.g. repeatedly sends messages to a named service should be eventually JIT-optimized into one that grabs the PID of that named service and messages it directly. Of course, if that named service dies, the process will crash—but like any other JITed code, that's just the point at which the JIT abort-traps back into the un-optimized codepath and re-runs the function. This can be generalized to an arbitrary degree; you can go so far as to imagine e.g. distributed Erlang nodes that pass bytecode to one-another and gossip about code-changes, in order to be able to JIT-inline remote (pure) function calls.
Say we've implemented your entire idea with JIT-optimized message passing. What would happen there is people learning ins and outs of particular JIT implementation and JIT-hacking being a required topic on job interview, just like today knowing millions of ways of hacking CPU cache is something expected from a professional (non-web) programmer (web programmers are expected to know there is something called 'processor' and that it doesn't like nested for loops; source - was a professional web programmer).
Anyway, while OTP does a pretty good job of papering over some of the local/remote objects difference and helps you keep the whole thing running even if something external breaks, I think it would be cool to go further in the direction of the ideas you just described.
I recently re-watched Yaron Minsky's "Effective ML" talk where (towards the end) he talks about making read-only and read-write types. That's where I thought Jamis was going with the state machine example: one could tell an object "yo, make yourself immutable!" (ie redefine your methods so that you can't change yourself). But in an OO world that seems more neatly achieved with subclassing. [To the extent of faking anything "immutable" in ruby"].
There's #freeze I suppose... and no #unfreeze. Which is perhaps sensible :) And #freeze only guards against new assignment to instance variables; it doesn't guard against an instance method mutating the content of an instance variable (def esquirify! ; @name << ' Esq.' ; end). So there's that. But I'm far from convinced...
Python has this "inner-methods" capability (with saner scoping) which is a great antidote for python's limited lambdas. But that's not a problem with ruby.
It's a neat trick (and nice to read something from Jamis again). Are there any sensible use cases?
class S
class << self
attr_accessor :singleton
end
attr_accessor :value
def initialize
@value = "xxx"
S.singleton = self
def S.new
S.singleton
end
end
end
a = S.new
# => #<S:0x007ff4a41532e0 @value="xxx">
b = S.new
#=> #<S:0x007ff4a41532e0 @value="xxx">
a.value = "zzz"
b.value # "zzz"
a.object_id == b.object_id
#=> true
http://www.tclcommunityassociation.org/wub/proceedings/Proce...
Somebody now shall goeth forth and implement Zork...
Bash:
$ foo()
{
bar ()
{
echo I am bar
}
}
$ bar
bar: not found
$ foo
$ bar
I am bar
This is simply because function defining is a kind of statement or expression with a side effect which must be evaluated. The side effect is global (a name is globally associated with a function). So if the side effect is in a function body, its evaluation is delayed until the function is called, and then its effect is still global.
In Python, an inner def will lexically bind a function, creating a closure. Python is not less dynamic than Ruby.
In Common Lisp, a defun inside a defun will behave similarly to Ruby; but if you want lexically scoped local functions, you use a different operator, namely flet or labels.
Scheme has a define which is lexical: it brings a lexical identifier into the scope for forms which follow.
Lexically scoped items, even in a dynamic language, in fact can be "cast in stone when you press the compile button"; they are cast in that stone which is the entire compiled environment of the surrounding function.
Does this work in Plan 9 rc?
Metaprogramming has a place in Ruby but for the examples in the article there are far more readable ways to implement it.
Readability > cleverness every single time.
Readability is in the eye of the beholder. For those unfamiliar with higher-order programming, maps are just a 'too clever' form of a for loop.
A clean and understandable solution is one matching the problem being solved in a precise way. In search for simplicity one can't forget that programming is a trade, and one should be expected to actually learn some shit.
In other words - just because you don't understand something doesn't mean it's "clever code".
This is hacker news, not "production code" news. Abusing things in unexpected ways is pretty much the definition of hacking.
Many people seem to be jumping on this article as if the author was suggestion a new programming pattern we should start using, rather than an interesting look at some of the lesser-known quirks of ruby. I'm pretty sure even the author would agree that triple nested method definitions are not something we should use for production code.
What I would say is that although we should have an extremely good reason to employ these techniques in production, I believe that every professional Ruby programmer should be able to understand them and/or figure them out in a few minutes.
It’s not like we’re talking about obfuscated C.
However, metaprogramming is a power that should be used wisely. When implemented unnecessarily it reduces readability (and probably performance) for no real gain.
The devise source contains some great examples of metaprogramming used properly:
https://github.com/plataformatec/devise/blob/master/lib/devi...