Please drop the SOAP | Hacker News Reader

Please drop the SOAP

thecoffman.com

121 pointsthecoffman61 comments

One of the things in life I really regret is suggesting SOAP on a project. I thought it was just a packaged, somewhat elaborated version of a semi-reasonable XML-RPC type thing --- and worse, that the idea that it was standardized meant that you could count on some kind of half-decent interoperability.

It was initially sold that way, but that was just the first of a parade of sales pitches, hilariously chronicled here, http://wanderingbarque.com/nonintersecting/2006/11/15/the-s-... --- endlessly shifting as if seen through a fun-house mirror.

SOAP is a nightmare.

soapUI is an application that will let you fiddle around until you get the calls right.

I recently did exactly this. I wound up not only generating the SOAP envelope by hand but also the HTTP headers:

SOAPAction: "http://...//SubmitDataform

What is the action I am trying to do doing in the HTTP headers?

It's this lack of discoverability that kills SOAP. I've never had any trouble parsing the response xml once I get my hands on it, namespaces etc are a non issue at this point. The completely non-standard format of the queries that makes it such hard work. With REST you just have to figure out a url. With soap you have to do that plus work out how authorization is handled, how to embed the procedure and parameters and then what namespaces to put everything in.

A good SOAP library lets you plug in a WSDL and returns an API object with all the methods of the webservice. There is literally no equivalent capability for REST. You shouldn't be doing any of the things you describe by hand.

You are vastly oversimplifying when you say "with REST you just have to figure out a URL". Try integrating with a service that takes only custom HTTP verbs with multipart/form-data but because there's no standard way to send an array in HTTP you have to append some parameter multiple times but in the right order. There's some vague attempt at using OAuth for authentication, but it doesn't work with any of the libraries you are used to. Oh, and the service returns a 503 some of the time, which (according to the rambling hacked up docs for this API) means you should retry your request in 10 seconds. No, it's not always trivial.

You're right that in theory a REST api could also be a PITA to use. In my experience though I've always been able to get the query I wanted quickly and easily. To be fair it's rare that I'm doing anything that isn't a GET but on the occasions where I have wanted perform an update It hasn't resulted in any of the difficulties you mention.

It's this lack of discoverability that kills SOAP

I thought it was discoverability which was SOAP's strong point, while ad-hoc solutions like REST which have the bonus of being much more lightweight/terse, lacks this feature.

As for non-standard query-format, with a good toolkit you point to the WSDL and you are done, all your RPC-calls pregenerated for you.

I know SOAP has some issues, that it's a verbose line-format and that recent WS*-additions add complexity faster than you can say "oh shit", but at its root, SOAP really isn't hard to work with if you have a decent toolkit.

If you are hand-rolling your SOAP-layer, you might as well start invoking your database trough telnet and handrolled protocol-code. It makes as much sense.

If the REST architecture is sufficiently mature, it should be entirely discoverable, and in a much more granular fashion than SOAP, without any tool more complex than curl. That's what HATEOS is all about. The difficulty is that saying that an interface is RESTful doesn't guaranee that: there's no naming difference between a service at level 3 on the Richardson Maturity Model, where this does apply, and one at level 1, where you can't even rely on HTTP verbs behaving predictably.

As to the existence of sufficiently good toolkits, that won't save you in the face of vendor bone-headedness. Talking to .NET from a Java stack is still problematic, where either toolkit on its own is apparently fine on its own territory. There's a particularly fine example in MS Exchange Web Services, where getting Java to talk to it over SOAP is so fiddly they're releasing a Java port of their managed API.

To be fair Exchange Web Services is a prime example (IMO) of how not to do SOAP. It's the philosophy of SOAP being "message-passing" taken to the extreme.

You have the SOAP messages which envelopes (sigh) yet another message-level architecture where every request, even at OO level means constructing request-objects with millions of sub-properties, all types fitting an abstract inheritance tree. Which ofcourse is also what you get returned after sending your request. So then you have to start analyzing your response object (again a message-object enveloped in a SOAP-message) and anaylze what class it actually is and the same for all subproperties.

I don't mind SOAP so much, but EWS can really go die in a fire. You can say whatever you want about SOAP in general, but EWS cannot really be used as an example of anything other than itself. I've never seen such a painful and convoluted solution anywhere else.

Even if SOAP had been ditched and the wire-format used had been some sort of RESTFul implementation, it would still be a clusterfuck to use. Blaming SOAP here is IMO a bit misdirected.

I could eventually come to ignore the fundamental screwed-up-ness of the API if it didn't also mess with the protocol itself; Exchange.asmx dropping out an incomplete WSDL is icing on the fractally wrong cake.

Anything can be easy to use if you have a toolkit to hide the complexity. The point is with REST you don't need the toolkit.

You Are Doing It Wrong.

Anybody that tries to interface with a SOAP interface by writing envelopes by hand is wasting their time and half-implementing the (long, terrible, complicated, blah blah) spec.

Find a good toolkit. Yes, the protocol is way over engineered but good tools can and do abstract that all from you. For instance, PHP's standard library lets you plug in a WSDL URL and consume SOAP services in two lines of code. (And because it is a good toolkit, it lets you spit out all traffic with one debug flag, unlike whatever terrible thing the author of this article is using.)

Most of what people complain about in these comments would be moot with good tools. You would probably complain about REST if you didn't have curl or Firebug and had to telnet all your test calls--that's equivalent to what most people here seem to be doing.

I don't know what wsdl's you were dealing with, but I met a fair amount of them who were not consumable by neither php-soap, python and even a java client while trying to read a .net one.

The reason people are writing their own envelopes in cases (like I did also) is because the tooling is not delivering up to it's promises. (Let alone that the server might not even understand the two-line-code genertated nusoap reponse because he interpretes some ws-* rules about namespaces differently.

The conclusion is: nobody wants to write his/her own envelopes.

Every time I here about regular Soap clients failing, it's always in regard to trying to consume a .NET service.

I don't know anything about .NET, but I wonder if there is something profoundly different that is going on behind the scenes, or does it just generate crufty WSDLs? Can anyone shine more light on this?

My problems were in fact with Java web services. There are different interpretations possible of the recent SOAP standards (or non-standards, as some are more de-facto than ratified) of smaal details as where to put type-info in the wsdl and if it is allowed to be redundant. This way sensible tooling is defied. Serving and consuming smaller services with light messages and shallow structure works mostly out of the box.

The going gets rough when people put half their strongly typed domain logic in the wsdl which has to be recereated on the other side. Hint: Collections != Arrays != Hashes

So, problaby not .NET but bloat related. Interoperability is a problem on all sides though.

Thanks for your info. I've tried (in the last few hours) to find a good toolkit to consume WSDL rather than write my own envelopes as you mention.

I'm investigating 3 different WSDL->JS libraries.

* Guru4 JavaSriptSOAPClient - Unmaintained since 2006

* Python wsdl2js - not widely tested or maintained

* Apache CXF wsdl2js - maintained, but doesn't seem to like the WSDL provided by my service provider.

Of the three, Apache CXF is the only one that looks promising, but fails to generate any JS. I'm trying to fix it, but I keep encountering more old-style web-services acronyms. All I want are so classes - maybe I'm doing something wrong, but I simply don't think it should be this hard.

If you're interested, the WSDL issues are detailed further in http://stackoverflow.com/questions/4827467/understanding-apa...

Edit: I've found a maintained version of the Guru4 implementation at http://javascriptsoapclient.codeplex.com that I'm going to try.

Autogen very often fails if the WSDL generator of your provider and the WSDL consumer/code generator are one and the same.

Trying to generate code using a Java toolkit from an MS wsdl? 90% chances of failure. Same the other way around.

This is one of the main reasons SOAP has been a catastrophic failure.

A protocol for interoperability that is vendor-specific? That's an oxymoron.

> Find a good toolkit.

I spent a few weeks trying to find a good toolkit for the big hairy nested ape of a WSDL I was trying to consume, and nothing could parse it properly and expose the methods in a sane manner.

I ended up doing just what the OP did - hand-crafted the request payloads in soapUI and turned them into templates that I could execute programmatically.

I really hate SOAP.

>> soapUI is an application that will let you fiddle around until you get the calls right.

> WSDL

In the OPs defense, it should be pointed out that SOAPUI is a WSDL client.

Three years ago I've had to interface with SOAP-based service. After trying various libraries for various languages - only to find out they don't support something (I actually remember lack of ad-hoc polymorphism, inability to understand the response format with some weird reference-like structures, and inability to emit some obscure Java datatypes whose names I forgot) - I've ended up with generating query XML template by hand and processing replies with quick-and-dirty PCRE-based parser. Sure, it's Wrong(tm), but I did it in a lunchtime and the code worked like a charm.

Sure, if I'd write in Java (I ended up with Perl5, as - thanks to CPAN - it seemed the most suitable for the task), all this SOAP stuff would be as simple as several clicks in NetBeans or whatever-Java-people-are-using. But I didn't knew Java enough to write everything I needed, I didn't like idea of Java SOAP-to-SomethingRESTful proxy, and I was quite scared of FactoryFactoryFactoryInterfaceActorProducerFactoryFactorySingleton libraries to dive into Java (yes, I know not all Java is Enterprise Quality, but system I had to interface with was extremly enterprisey).

Debugging SOAP stuff is a complete nightmare, but trying to figure out and implement somebody's RESTful API that has 100 different types of objects is also a hassle, since there isn't code generation that can help you.

For tiny one-API-call tasks, I'd much rather use a JSON REST API that I completely understand. When I'm trying to interface with a complicated system that has a million different objects, I'd rather spend an hour figuring out the tools, then rely on generated code so I can take advantage of IDE auto-completion instead of digging through the documentation to remember the params and return value. It's the in-between of these two cases that's the devil, especially since big nasty APIs always start out small and simple.

I'm pretty sure everybody read this piece: http://harmful.cat-v.org/software/xml/soap/simple

Urgh. I had exactly this experience trying to get the Exchange Web Services API to work. That's worse, because the author's solution, dropping any pretense at using a notionally "pluggable" API and just debugging with Wireshark, won't fly over HTTPS.

Try it with Fiddler.Should work fine over https.

No Windows boxes round these parts, or I would have. I tried WebScarab, but without the server certificate it was a lost cause (unless I missed a trick somewhere).

You have to generate a certificate for the proxy (WebScarab) and have the client trust it. The proxy will then act as a man-in-the-middle, presenting its own certificate to the client. See:

http://www.owasp.org/index.php/Generating_Custom_SSL_Certifi...

Now I know what to do if I ever find myself in that situation again. Thanks.

Charles proxy works on OS X and Linux.

Thanks for the fiddler tip - I also use Wireshark for most of my debugging needs and https always causes problems.

I'm going to check fiddler out too, but I've been using Charles proxy for the same thing if you need another way to see SSL.

Perhaps the fact that I was surprised that the article's author didn't immediately think of Fiddler is evidence of how exclusively I've worked with web services via .NET. Personally, I've had a great experience with SOAP from within Visual Studio. Once the service reference is established, all coding is against the strongly-typed code-generated classes. If I must, I use Fiddler to debug. My experience has been positive.

Cannot sympathize with you enough. Had to go through this painful experience last year. I steadfastly refused to work with it again and we solved the problem in a roundabout manner. I'm sure .NET/Visual Studio experts think this a cakewalk, but for me there was no cake.

People still use SOAP? Yikes, sorry.

Sad part is its not just "People" Google's AdWords and Microsoft's AdCenter are both SOAP. Par for the course from Microsoft, but I'd expect better from Google.

Yeah, I have to work with the AdWords API and it sucks just as badly - the documentation also lags behind the implementation too.

The super annoying thing is worrying about WSDL, XML namespaces and xpath for something as simple as getting a count of alerts.

Their client library is decent, for PHP anyway so you mostly get SOAP abstracted away. Up until you get an error then the usual problems are there. The underlying code in the client library is super bloated though, like what you would imagine from visual studio endpoint type stuff in the article. It may be that the core adwords developers are using Java and an old school SOAP interface makes sense to them. They certainly had a chance to change it recently with the new version.

in Python, they make you chose between ZSI and a beta soappy release. Their site has a getting started video which mentions running a config script they no longer distribute. most docs are using a different module name. Oauth isn't implemented at all.

In JS, you have no client library so you're required to roll everything from scratch.

PHP has oauth, more demo code and better docs, I've recently been using it to build a JS lib for a work project.

The thing is, rather than maintain one client library, half maintain two others, and have a bunch of unofficial libraries, if they just made and documented a REST interface everyones lives would be easier.

Yeah, unfortunately I don't think it has a lot to do with the guys who have maintained the client library, I think they would agree.

Big problem is at the moment that PHP has Oauth for pretty much everything except report downloading, which is essentially separate from the rest of the API.

Hoping they fix this before I launch, as most of the good of OAuth is negated if I still have to collect a username and password for report downloading.

Soap isn't really good, but its not just MS who pushed it. In the 90s it was endorsed by giants like IBM and Oracle

Completely right - its not a Microsoft problem as much as its an "enterprise" problem. IT Managers love things like SOAP.

And they should. It's easy to turn a WSDL into an exhibit for a contract, because it is a description of an API that can be converted to human readable and machine readable formats. There is no equivalent for REST, and most REST services have a loose interpretation of what it means to be "RESTful". Try doing serious, paid-availability integration with REST and watch the managers tear out their hair and the lawyers weep.

Not that I'm a big SOAP fan, but the author seems to be seriously confused:

"For starters you have to point Visual Studio (sigh) to the service reference, at which point nine thousand (yes, that’s nine-zero-zero-zero) lines of code are generated."

"Not a descriptive error, or an HTTP status code, or anything that could be used to track down the problem but a “SOAP Error.”"

"Long story short, in order to inspect SOAP messages you have to write a class that implements the ICLientMessageInspector interface."

These are supposed to be problems with SOAP? To me they look like a crappy implementation.

The conclusion is also crappy:

"In my mind SOAP’s biggest failure is what some people consider to be its biggest success – it abstracts away all notion of the underlying HTTP model and replaces it with crutches like code generation."

Jeez, man - SOAP has problems, but code generation!? It's not a problem with SOAP but with the way your environment integrates with SOAP. Code generation over any protocol would create the same problems and there's nothing about SOAP that makes code gen inevitable.

I have to say, REST is just a lot more straightforward than SOAP. Although there are numerous advantages with SOAP, I always cry myself to sleep at night when I find out that I have to use a SOAP API to complete a part of a project.

Actually the guy is actually complaining about consuming a WS from WCF (important distinction because consuming it using the default WS stack from .NET is a much better experience). Besides that...carry on praising REST.

It was just a bad SOAP api. You can write a bad and cumbersome REST api as well if you want, and MS would certainly be capable of doing so. SOAP is not really the problem here.

While this problem was for sure exacerbated by a bad API design (and I've seen pretty horrific REST ones as well) the fact that all of the underpinnings are abstracted away makes it very difficult to debug in any sort of depth when things go wrong. That's the ultimate point I was getting at - I'm fairly new to blogging so it is very possible I'm not conveying my point as effectively as I'd like.

SOAP is the problem here. Specifically, it's the architectural decision to hide the protocol details which inevitably ends up biting everyone who uses it, because it's a textbook leaky abstraction.

"Specifically, it's the architectural decision to hide the protocol details which inevitably ends up biting everyone who uses it"

Sorry I don't agree.

Firefox for example hides the protocol details. You don't need to know DNS, TCP/IP, HTTP to browse the web.

It's the same for SOAP. If with "hide the protocol details" you refer to this part of the post

"I got an error. Not a descriptive error, or an HTTP status code, or anything that could be used to track down the problem but a “SOAP Error.”"

you may not be aware that the SOAP specification has SOAP exceptions. And they are actually quite easy:

<soap:Fault>

  <faultcode>soap:Client</faultcode>

  <faultstring>Please specify your birth date</faultstring>

</soap:Fault>

He just used a bad SOAP api that returned an useless error.

There are many other things about SOAP we can hate and that need to be fixed, but in this particular case SOAP is not at fault, and REST could very well be exactly as bad.

> Sorry I don't agree.

> Firefox for example hides the protocol details. You don't need to know DNS, TCP/IP, HTTP to browse the web.

I don't think that example supports your argument. Quite the reverse, in fact. Firefox doesn't hide that much of the details. You can still see the gory details of URLs and HTTP requests surfaced quite clearly, and the experience of using the web is that much richer if you know what they are and how to manipulate them. Firefox doesn't actively prevent you from doing so if you want to, which it would if it was trying to present a sealed abstraction.

As regards errors and error reporting, I don't particularly want to get into REST v SOAP, because that's tired ground that's been trodden a thousand times over. However, REST has an advantage over SOAP here in that it uses HTTP error codes. An app which returned a 200 response when it couldn't find something would never be described as RESTful, whereas a SOAP app can get away with anything as an error response. It's simply not possible for a REST API to be bad in that sense where it demonstrably is for SOAP.

>Firefox doesn't hide that much of the details. You can still see the gory details of URLs and HTTP requests surfaced quite clearly, and the experience of using the web is that much richer if you know what they are and how to manipulate them. Firefox doesn't actively prevent you from doing so if you want to, which it would if it was trying to present a sealed abstraction.

Contrast with IEs 'friendly' error messages by default. Instead of an easily diagnosed 'page not found', or 'address not found', or 'server error' you get 'something went wrong' and then a list of always incorrect guesses as to what the problem might be and a confusing list of irrelevant things you can try to 'fix' the problem. I bet somebody got a bonus for that feature.

Most of the OP's problem was the fact that there was no simple way to JUST INSPECT the soap messages going back and forth. You can say that is more a problem of his tools than SOAP. But then XML (and especially SOAP) is so friggin' ugly that of course toolmakers are going to want to abstract into invisibility.

That said, SOAP really is more complex than it needs to be. What makes it worse the ironic name : "SIMPLE object access protocol".

As I understand it it's not a problem of the API but the nightmare of layers upon layers upon layers together with the promise that this will be hidden from you. (Not that having to fumble out what that REST service is actually sending back to you sounds that promising)

But the last three SOAP services I had to swallow ..err.. consume, leaked their abstraction over my desk and I had to start to xpath my way through responses from a Java service after having curl'ed my handwritten requests to the other side. This was not due to a problem with the published API but with the protocol annoyances as such. (e.g.: your OrderClass is not my order_class and your Array() is not my [] )

"I had to start to xpath my way through responses from a Java service after having curl'ed my handwritten requests to the other side."

So you actually used SOAP like REST with XML :)

Yes, but he didn't want to. If SOAP lived up to his expectations he's just be running autogenerated methods.

To give a slightly different spin to the discussion: does REST solve some of the problems SOAP addresses? First there was REST, now closely related to HATEOAS... I see the acronyms are coming.

How to discover the meaning of data in the absence of a WSDL/RESTDL? Will we have microformats with links pointing to ontology-servers? Wat if its JSON?

I wonder what REST looks like five years from now, when for instance, IBM and M$ become involved in a standardization process ...

I don't like Soap for the same reasons everyone hates it. Having said that, in the Java space there are plenty of tools that turn the Soap experience into a bearable one. With CXF, for instance, is quite trivial to generate client code out of a WSDL file. The generator binds the request/response payload to Java beans so the developer doesn't have to understand the intricacies of the protocol. Debugging is also simple.

Diss on SOAP if you want but it does provide good error support for working with transactions in Java EE 6, ESB, etc. If you need to use infrastructure that works with SOAP, then use SOAP and stop whining.

If you don't need to interop with existing systems that use SOAP, then sure, a REST style makes sense.

Since the op is using WCF for calling the web service he would have a much easier time figuring out what is going on if he turned on WCF tracing. Every step in the pipeline including the messages sent on the wire and timings are available.

Once you know the secret platform handshake these enterprise tools are pretty easy.

Edit: fix annoying iOS auto-correction.

SOAP in visual studio might be a pain, but its really not that difficult when working in PHP:

http://php.net/manual/en/book.soap.php

I've implemented both servers and clients using SOAP with low to no hassles.

    Bad implementation != bad spec

edit: appears to be back and working fine.

Embarrassing - despite my best efforts it appears as if my little linode couldn't handle being frontpaged - here's a cached entry for now:

http://webcache.googleusercontent.com/search?q=cache:UehgGNU...

The blog entry is cached as a static page hourly and served via nginx as such - this load shouldn't have killed it. Will be interesting to dive into.

LOL, Wireshark. Nice.

Reply on news.ycombinator.com