How We Are Improving Performance
blog.imgix.com
blog.imgix.com
We switched to host our own thumbor (open source), and couldn't be happier. We pay around a quarter or less than before (even with failover in place) as well.
We really wanted to use a hosted service. We're not keen on hosting stuff out of our core business. But in this case it just didn't work out.
EDIT: link to Thumbor https://github.com/thumbor/thumbor
I was actually surprised there were no hosted thumbor services out there (I found one in Beta I think at the time I was looking, but we weren't sure how stable it is). Maybe because it's so easy to self-host that nobody bothers offering a hosted solution.
We do have some interesting solutions in the lab for more elaborate compositing amongst other things. Part of the challenge there is to stand up a new API interface, since the URL API gets pretty cumbersome for coordinates and tons of composites. We have some interesting ideas there, but it needs to bake a little while longer.
It's also a challenge to get that new stuff out while focusing heavily on our core areas, but we have to do both. You'll start to see more exciting new things ship from us over the next few months.
If you have a solution that works for you, that's great and best of luck. If you do ever feel the urge to check in on what imgix is up to, we would welcome it. Feel free to reach out to me directly any time (e-mail in profile) and I'll work with you to get what you need.
I think your support could have been more responsive though, and communication improved. You guys could probably detect that we weren't happy and about to leave. Reaching out at that point could have made a difference perhaps...
We did genuinely want to work with imgix, but those hurdles left us looking elsewhere. These things happen all the time. Competition is a good thing overall. I'd like to hope that our feedback helps in some way. Even if it's hard to hear.
The unfortunate reality this time around is that we got overwhelmed and had to focus on doing work that had the most impact to the largest number of customers. That meant that more personalized approaches to customer retention kind of slipped through the cracks. This can't happen again, and I'm adamant that it won't.
I see image processing services as an important area for innovation and growth on the web over the next several years. It doesn't mean that imgix has to get the business of every site on the Internet, but fundamentally I believe that anyone who cares about how their content is presented should use a service like imgix. Doing image manipulations by hand, or serving non-optimal formats or sizes to browsers is going to be viewed as serving your site without using a CDN: it just isn't the state of the art anymore, and the benefits are worth it.
I welcome competition from other companies who run competing services, or build-it-yourself solutions like thumbor, because that means we really were on to something: the market is real and these services provide real value to customers. I want imgix to be at the front of the pack, and that means doing the hard work every day to make sure we're there.
This feels a lot more like the latter, and it's wonderful to see. I'm sure it's a lot harder to write, but it shows a heck of a lot more integrity.
For instance, losing three backup generators in ten minutes (as happened to someone else a while back) might be bad luck, but I'd want to see the maintenance logs and talk to the vendor. Because what's much more likely is that they hadn't been maintained properly, or there was an manufacturing defect in that run and they all came from the same batch (see also, clustered hard drive failures).
If their upstreams have had intermittent problems before, it was only a matter of time until they both had problems at the same time. And capacity planning is one of those strategic decisions that gets bland on the engineering team, who never get what they need because they're a 'cost center', which is a very different and highly damaging way of thinking about the cost of doing business (intrinsic costs).
> If their upstreams have had intermittent problems before
Sorry, this might not have been clear. We had never had major issues with any of our upstream providers before -- beyond scheduled maintenances there had never been a service interruption over a number of years.
There's almost always the risk of coordinated failure, no matter how many levels of redundancy you put into place. The cost gap between 99.99% and 100% is humongous for this reason.
From these failures, we learned some valuable lessons. One of our transit providers was acquired a year or so back, and it's apparent that things have dramatically changed in how they operate their service and respond to issues. We won't be continuing the customer relationship there for much longer unless we see positive steps taken. Another transit provider has been much more responsive and the conversations have been more productive -- we can work through the issue we observed and get back to a positive situation there (I think).
And then finally, we need more diversity. We turned up another circuit yesterday, and I'm still working on adding another 1-2 providers in addition to more peering or direct connect situations. These take a bit of time, but it's work we need to do before we actually require it, because by then it's too late. We're working as quickly as possible to get this part of our infrastructure on rock solid footing.
> capacity planning is one of those strategic decisions that gets bland on the engineering team
Absolutely. We were doing capacity planning, but it wasn't anyone's idea of a good time.
We are correcting this with the resources we have today, and we're going to do a better job in the short term. In the long term, this is a non-negotiable core aspect of someone's job -- either already on the team, or a new role that we create. It can't be a secondary, rainy day kind of task you take on time permitting and it also can't be a task that we're not well equipped to handle on a regular basis.
The driving reason that led us to start imgix, and then spend years of our lives growing and operating it, is to empower our customers to build awesome stuff on the Internet. We have to work hard to keep up our end of the bargin, and we also have to forge our partnerships and relationships with customers in a million little ways. I hope that through adversity we've forged some new relationships with customers in the way we've communicated this time (and how we will continue to communicate).
As an aside, I was personally impressed by how one of our vendors handled a service issue they had last year [1]. That was a definite inspiration for how we handled this communication.
[1] https://ns1.com/blog/how-we-responded-to-last-weeks-major-mu...
Disclaimer: I am currently a customer who had financial loss because of them.
Their support (after few days said), they don't support Shopify. Their whole pitch was; we are Shopify compatible; when we signed up.
I'm sorry that you had this experience with imgix. We have had challenges in correctly addressing support issues during this period -- support has been swamped. We've expanded that team, and we're continuing to do so.
You're right, we didn't tweet this blog post, but we did e-mail it to all customers with activity since Jan 1 2017.
In this case, we didn't properly convey that the Shopify integration guide we provided wasn't an official integration with Shopify. It was meant as a best effort "here's how you can do it" sort of thing. I requested that we take it down because it wasn't working due to a change on the Shopify side which prevented us from purging images.
I'd love to talk with you directly to try to make this right for you. If you'd like, my e-mail is in my profile.
"Support is on-call during regular business hours, Monday - Friday, and premium customers can request SLAs for support."
This, while other users complained to me via twitter/email that they were also not getting any valid responses. I would not send those words while my service is not working as expected.
I'm in the process of moving out of Imgix esp. after your support said they would be happy to cancel my account if thats what I wanted.
The pushback from your support made me setup a production level thumbor instance which will go live next week.
Thanks for your reply but I wish you had replied to https://twitter.com/troysk704/status/841158793287278593 sooner.
Best of luck with your new solution.
A critical piece of our network infrastructure failed after 3 years of correct operation in a way that proved difficult for our network engineering team to troubleshoot.
Which piece of infrastructure? How did it fail? Why was it difficult to troubleshoot? Can you do anything to prevent this type of issue in the future?
We observed new traffic patterns with significantly lower cache hit rates than our historical median, and it took us some time to determine whether the source was abusive in nature or a legitimate new customer use case.
What were the new traffic patterns, and why did this cause lower cache hit rates? Why did it take longer than expected to determine the nature of the traffic?
> Which piece of infrastructure?
In this case it was a top-of-rack switch. That rack had a few of our external DNS resolvers on it. Normally this wouldn't be a problem, because we deploy services across multiple racks to prevent this kind of failure mode.
In this case it was a bit of a problem, because this was the first rack deployed at this site and it turned out to not conform to the standard configuration across the rest of the racks (because it was what we bootstrapped the site with).
So two things: it was tough to get into the device to troubleshoot it (we wound up using the serial port infrastructure deployed for this purpose), and it had a service impact even though it shouldn't have (we have since migrated DNS resolvers out of this rack and have scheduled a future maintenance to get this switch's configuration corrected and ugprade its OS).
> New traffic patterns
This is a bit tougher to dig into, but essentially it was a needle-in-a-haystack kind of situation. Customers can use any imgix URL API parameter they like -- some of these use cases get pretty complicated for the backend to handle. Think of the watermark parameter stacked with another URL that is also an imgix URL with another watermark parameter, five layers deep. These sorts of operations take a lot more rendering resources than a simple ?h=600&w=600 operation.
In this case, we observed an influx of these sorts of more difficult operations. We have various logs and metrics sprinkled throughout the system -- we use Prometheus, Kafka, heka, BigQuery, Grafana and a few other systems to collect and present the data we need to run the service. We also issue unique id's per request to track their path through the system. What we don't have -- and need to -- is one end-to-end view of a request's path through the system and the system's capacity and performance at each point in our stack.
It turned out that for some amount of the increased rendering traffic we saw, it wasn't that we suddenly got more requests. There were simply many new permutations for each original object. That lowers the CDN cache hit rate for a period of time.
The other thing that comes to mind (and I think I'm forgetting a third type here), is that some of our request parameters require normalization so that they can utilize the same cache object. Think of parameters like dpr, which is a floating point number but realistically is only useful up to a few decimal places. dpr=1.33 and dpr=1.3333333333 are actually the same image, but they would have different cache keys and require two renders that are effectively the same object.
We normalize the dpr parameter down to three significant digits. What we found is that this sort of normalization was necessary for another parameter as well.
When you have an incident, especially something that hurts your customers beyond just a service outage, you come clean, fast, and as completely transparent as possible.
We posted the blog post yesterday (Friday) afternoon, after getting the copy ready Thursday evening. Friday was spent acquiring the right customer e-mail list (a little harder than it sounds and something to improve in the future) and taking one last pass over the post (among other tasks we're all doing to actually operate the service, communicate to individual customers, etc.).
It's super important to be open and honest and fast in your response, but it's also important to communicate correctly. The post in its current form is meant to be transparent and apologetic but also show that positive steps have been taken and results are being seen now. If we missed the mark on transparency, or honesty, or the hints of positivity, it wouldn't have been the right message for where we truly are at today.
I'm completely open to writing a more in-depth technical post, but that would have compromised our timing to get this post out on Friday. That's why we went with the post in its current form -- anything else would have taken longer to get to our customers.
It's something that I'll think about more this coming week and hopefully get out soon.
Isn't that precisely what they are doing here? This looks to me like a case of a company doing an excellent job of clearly communicating about an incident, at least from the point where the post was first published.
> We operate our own hardware, run our own datacenters, and manage our own network infrastructure.
This seems insane to me. Although I don't work with image processing beyond "saving for web" in Photoshop, so I could be wrong. Why would they not use AWS or any number of other cloud providers where capacity planning is handled for you?
The main reasons are twofold:
A) cost. I haven't done the math yet in 2017, but my recollection of the difference circa 2015-2016 between GPU instances on AWS and our solution is a COGS that's about 3x higher on AWS. GPU instance prices have come down a bit since then, but they're still very expensive and somewhat supply constrained as well.
B) Technical frameworks. We'll reach an inflection point this year or next where we've added or changed so much of the rendering pipeline that we've basically written it from scratch, but until then we're still reaping the benefit of building on top of CoreImage. It really benefitted us early on in getting a MVP faster, and it's continued to pay (slightly reduced) dividends over the course of the company's life. At some point the dividends will stop, and then we'll move on.
I understand the skepticism around Mac Pros. There are challenges there -- it isn't my most favorite solution of all time. It is a practical solution for us though. I have no room for computer religion or deciding to do things because they're cool or shiny or new. Anyone that's ever spoken to me for a few minutes in person about technology stuff can attest to that.
The Macs don't have IPMI, for example. That sucks but we do have power outlet control and a network installer as a way to back into "out of band management" for them. They're largely stateless, so it's a tolerable solution.
They do run a Unix-like OS (thank god), and they do represent a good price / point ratio for image rendering hardware. We could do a little bit better with Linux servers and GPUs -- but the upside is only around 10-20%, and there's still engineering work to do there. It's on our roadmap to explore this more fully and maybe start taking more concrete steps in that direction. For now, OSX still gives us more than it takes in terms of cost.
Sure, image optimization is CPU intensive, but their use case is super bursty. When the source image changes, you do multiple optimizations (convert to WebP, lossless JPEG, lossy JPEG quality change, etc), cache the results and you are done. The ratio of "optimizing image" to "serving optimized copy" must be insane.
Dedicated physical hardware feels like a waste.
> why not use elastic infrastructure?
This would have been beneficial (to a certain extent) to solve the over-capacity issue that we faced, but it wouldn't actually be of that much help in an under-capacity scenario. None of our machines are ever idle -- it's simply a matter of how long the work queue gets.
> The ratio of "optimizing image" to "serving optimized copy" must be insane
Yeah, but the serving an already rendered image doesn't traverse our image rendering stack, as you might imagine. That content is cached at the CDN edge (or failing that, within our rendered output cache). We don't re-do work when we don't have to, that's an inherent part of how we keep the cost of operations right and provide our customers with a service of this complexity at this price point.
Sorry that you experienced these issues. I appreciate your understanding.
Please feel free to reach out directly to me (e-mail in profile) if there's anything you'd like to talk through.
I'm optimistic that you will have a better experience going forward, but I can guarantee you that we are doing everything possible to provide the service to you and we will do a better job of communicating in future.
I'd be curious to hear exactly what they did around this. I recently worked on some gif encoding, and was surprised that there's actually quite a tradeoff to be made between performance and good color palette choices.
I don't have the specific details handy, but I can confirm that it's a bit of a tough one. It's actually much easier to do work in pretty much any other format -- the very reason why people use GIFs (universal support) is also what makes them tough to work with (the spec is old and crusty and from 1989).
I'd like to do more of these sorts of posts -- talk about the challenges with GIF encoding and how imgix has solved them. Probably not this month or anything, but I'll try to get it on the content roadmap for April or May.
Apparently not: https://blog.imgix.com/2015/05/08/racking-mac-pros-hardware-...
Perhaps I am missing the competitive advantage of using OSX to resize images, but it sure seems like the shortcomings are obvious :)
It's an understandable direction if your business is at the "Well, I have a few of these mac pros sitting around, I wonder if I could make a living at this" stage.
I've touched on this elsewhere, but here's the crux of the problem:
Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch.
The chassis that we co-designed and had made works out to a few hundred dollars per machine, since we aren't buying in giant quantity. When you talk about servers that run 24/7 and cost between $5k-25k per box, that's not the dominant expense by any measure. It's also been picked up by some other companies to use, which is always nice to see -- we aren't keeping it as a proprietary solution, anyone can buy it from our vendor.
Either way we decided to go, we would still be solving both the image rendering and machine racking / operation problems simultaneously to different degrees. As it sits now, we have about 4-5x the number of infrastructure and imaging engineers as we have datacenter managers / technicians on staff. As it should be, I think.
This design decision is actually part of our advantage, in that it allows us to deliver more functionality to our customers at a lower price per unit on the backend. There's always room for improvement, and we continue to iterate on it -- some day the Macs will probably be gone from the datacenter, but only when it's the right move to make.
The trashcan Mac Pro has very outdated Xeon E5620 CPUs and it doesn't seem like there's a lot of upgrades in the pipeline. I can even see Apple stopping the sale of these in favor of iMacs.
I think any 'regular' sever with e.g. an Intel Xeon E5-1650 v3 be about twice as fast for basically the same price.
I'd assume that image manipulation is mainly a CPU bound issue.
> Racking servers, even if they're cylindrical, is actually much easier than building an entire real-time image rendering pipeline from scratch.
Scaling a service based on Mac Pros is probably much harder than starting to port pieces of a real-time image rendering pipeline to hardware that is a lot faster and more readily available. To be honest, for the price of the Mac Pros, you can probably run a lot of CPU hours in AWS / ... and skip the whole rack-n-stack.
That being said, I'm not the one running an image rendering business, so I'm just armchair quarterbacking :)
It is annoying that Apple has let the current Mac Pro rot on the vine to a large extent. This is something that certainly gives me pause and will be factored in to the future direction of our rendering pipeline. There may not be much of a future for image rendering on Macs at imgix, but we have to factor in all the variables when deciding on our future roadmap.
The difference between a v2 and v3 CPU definitely isn't double performance, by the way. An E5-2697v2 (the top-of-line Mac Pro CPU) is a 970 on SPECintrate, and an E5-2697v3 is a 1240 on SPECintrate (we track other measurements, but that's the quickest comparison I found). So it's better for sure, but it's also 15W higher draw and has 2 more cores to help grow those numbers. Per-core performance on SPECint is only 60 vs 64.
That said, in a perfect world I would love to use v3 or v4 Xeons on our image rendering hosts. It just wouldn't be a night-and-day kind of difference that's worth throwing our existing pipeline out over. Have to be careful not to toss the baby with the bathwater here.
The real performance issue for us this time around was adding machines to overcome certain scalability constraints, and optimizing certain code paths that were prohibitively expensive.
I think the point here is that your roadmap may lead off-road once Apple finally does to the desktops what they did to the headphone jack on the iPhone(ignore the outcry, do whatever they bloody-well please).
I think this is still a penny wise, pound foolish decision.
I am sure it is less work to figure out how to buy and rack some trashcans, vs writing an entire image pipeline for 1.0. I get that.
I also don't know your full specs. There are tons of FOSS image libraries out there. What % of your workflow do they cover? Are you sure you are estimating taking something like Pillow and adding in missing pieces, vs writing an entire image library from scratch? Things like basic filters and image resizes are a solved linux problem for a long long time.
Hardware lock-in always has a price. People that build on AWS exclusive things like lambda do the same thing, to a smaller scale. It seems like you are paying this lock-in price, and will continue to pay it. Perhaps it is the right decision, perhaps not. But I cannot image locking my entire company into the whims of a consumer trendy product line, when there is no fundamental need. I would look really hard at hiring some devs to build out a proper rendering engine...
However, there is no off-the-shelf solution that solves the same core problem as the imgix pipeline. Resizing or format conversion, sure. That plus the types of graphical operations that Photoshop supports? Less common. Now do all of that in one pass in 30-50ms, streaming to/from the GPU, with the ability to easily add new workflows and operations? It's our core business, and we own that workflow completely inside of our stack.
We built on the foundation of CoreImage and that gave us a leg up -- we can take most of what we've built, replace the foundation and keep going on another platform (such as Linux). The main challenge is that there is no equivalent technology to CoreImage on any other platform. PIL (or Pillow) are adjacent technologies, but I can tell you from experience trying to operate a PIL-backed image thumbnailing service at scale it is not a large scale solution. Better than ImageMagick for sure, but it still has limitations. Even something like RenderMan is not tuned for this kind of workflow -- it's awesome at what it does, but latency and scale are not its central concerns.
There are risks to imgix's business, and this is one of them, but I'll be pretty surprised if it sinks us. We have some pretty sharp cookies working on our rendering platform, and while success on that front isn't guaranteed it's entirely within the realm of possibility.
imgix does utilize a partner to handle some of our image delivery components. We're a customer of that CDN in that we pay them money for the services they provide, but we're actually more of a partner in that we have done considerable integration work and we work closely with them to build what we need to deliver our product.
We talk with them frequently, and there are pieces that aren't just off-the-shelf that we utilize to provide the integrated service we sell to our own customers.
To be clear though, the issues we faced that are discussed in this blog post are not related to the CDN or delivering already rendered images to end users. It's around rendering new images, which doesn't involve the CDN.
This is without zooming (100%), 120 DPI, Windows 7 x64.
Please use standard fonts and don't overengineer, font hacks like this will never work as expected across all platforms.
I'll prioritize rolling out the fix for this particular issue next week.