Hacker News Reader

Comment by kentonv | Hacker News Reader

Hi folks! This is my project at Cloudflare -- which, entirely by coincidence, I joined exactly one year ago today: https://news.ycombinator.com/item?id=13860027

Happy to answer any questions.

View on HN

This is awesome.

I guess the uses are endless. Debug some webhook endpoints by duplicating the request to a private requestbin - granted this isn't the right level to do this but sometimes we're working with legacy setups. Add missing json fields that an API broke for some client - again not ideally level to handle but sometimes we bandaid it. Send an API auth spammer gigs of random garbage because chances are their shitty client doesn't have any limitations - granted after 14.9 seconds you'd need to subrequest to your own garbage generating origin - or a 10gb garbage file somewhere

My uses would mainly be live debugging where we don't have a perfect stack (i.e. nearly all the time).

I hate to say this, but you can send more garbage than that. Subrequests have to be begun within 15 seconds, but they themselves can stream data for far longer. We actually support the TransformStream API to allow you to stream your garbage back and not keep it in memory as well.

Beyond that, we support the WebCrypto API, so you are welcome to generate a near infinite amount of cryptographically secure random garbage, and we'll even help out with some from our lava lamps [1].

1- https://www.fastcodesign.com/90137157/the-hardest-working-of...

I can't even load the page, so your miles ahead of me!

If anyone at Cloudflare cares to take a look, here's what I get:

> Server Error

> Our network is literally on fire

> Ray ID: 3fb0bf524995797f

> Your IP address: 172.58.40.122

> Error reference number: 502

> Cloudflare Location: Seattle

2nd try had an Error reference number of 524, with everything else being the same.

Shenanigans like this are why I stopped using Cloudflare on my websites, better to have something I can debug and avoid a party MitM all my connections, than to gain a fleeting bit of "reliability" or "security" that Cloudflare supposedly offers.

I believe "Our network is literally on fire" is an error from Ghost.io, the platform we use to host our blog.

Shouldn't your edge be caching this static article? I don't see why each page needs to be custom generated...

It does cache, yes. But there's a TTL on the cache since we want to be able to make updates to the page if needed.

The FAQ says that these are still in beta and not production ready - is that still true or has that page not been updated with this announcement?

Also, it says 50ms of cpu time, 15 seconds of real time as limits. When developing, are there easy ways to get measurements of how long things are taking? I can imagine 50ms on my MBP may not be the same as in production - either faster or slower - but I wouldnt want to get to production to find out.

https://developers.cloudflare.com/workers/faq/

> The FAQ says that these are still in beta and not production ready

Oops! That's outdated. Let me go fix that right now...

Workers are already being used in production by customers large and small today.

To answer the second part of your question, we are working on better ways to expose the resource usage of your Workers. That said, the vast majority of Workers consume less than a single ms of CPU time, so it is almost never a concern.

Hey, congratulations! Definitely going to play with this this weekend.

What are my options if I build something complicated using these workers and for whatever reason I need to stop using Cloudflare? Or if I want to write something open-source that people can deploy either on Cloudflare or on-premise? Is there a reasonable way to emulate this functionality (possibly with less sandboxing) on a self-hosted web server?

Also BTW you should update https://cloudflareworkers.com to note that it's live :)

> What are my options if I build something complicated using these workers and for whatever reason I need to stop using Cloudflare?

Great question. We chose to implement the W3C standard Service Workers API in part to give people some flexibility here. E.g. depending on what your Worker does and who your clients are, it might be possible to push your same Worker code to the browser as a regular Service Worker. My hope is that other services choose to implement W3C standard APIs in the future, rather than everyone doing something bespoke as they do today.

I believe there are also some Node.js-based Service Worker harnesses designed for server-side use, though admittedly I haven't tried any myself. It would be cool to see such things developed further.

> Also BTW you should update https://cloudflareworkers.com to note that it's live :)

Happening as we speak. ;)

Some immediate uses that spring to mind involve storing data as strings. An example is rendering templates from JSON or a more resource-friendly serialization protocol. What are the limits on the file size of the workers?

Also, regarding resource usage, is this memory usage exposed at all to the worker via an API? I'm thinking there may be applications where it would be useful to cache fetched resources but without hitting the memory ceiling and having the requests die with the worker.

The most common way to cache fetched resources is to simply fetch them and let the Cloudflare Cache do the hard work. There's no ostensible limit on how much you can cache there, but rarely accessed things do of course get evicted.

We haven't exposed memory information to the process yet for several reasons, one being it's rare to see a Worker that actually needs anything close to the memory limit (128mb). When people do run into that limit it tends to be in error. We would love to see what you can build that does do creative state management such that it would need that information though!

Are there any extra costs related to outbound requests one might make from a worker?

Nope!

One more question. Whats to stop this being used for amplification DDOS? Register a stolen CC then launch x100 parrell http(s) subrequest for each incoming request to your intended target, possibly adding large random payload (well as much as you can generate in 50ms minus time to setup 100 subrequests with fetch API)

If you do that, it won't actually send very much traffic to the victim site, and you'll find your zone suspended pretty quickly. :)

As with any anti-abuse measures, though, it's best if we don't describe in detail what steps we've taken to prevent this sort of attack.

I don’t agree that it’s best you don’t describe anti-abuse techniques. I’d be impressed if your manager agrees. I’ll argue points if you like.

Can you talk about topology relative to ingress and origin? Eg. is execution pinned in select PoPs per ‘region’ or all? Unrelated, I also wonder about retention of edge cache, is it purged via LRU or similar?

Personal note, I was excited to get the early access email, feels like an excellent offering; I’ll be flexing it soon, have some ideas—been getting an HLS system online in the past month using RasPi+CF without a hiccup (720p@4500kbps) and it’s open source/libre, comparable to YT/Twitch—info incl. design/ lectures in profile.

Also read above, congrats on what must’ve been a good year! Speaks well of your org that you’ve been able to deliver and talk freely.

> Can you talk about topology relative to ingress and origin? Eg. is execution pinned in select PoPs per ‘region’ or all?

Nope. Every Worker is distributed to every location. Requests are handled at whatever PoP they arrive at.

> I also wonder about retention of edge cache, is it purged via LRU or similar?

TBH I'm not that familiar with the inner workings of our HTTP cache, but I would assume so?

Unless to the cache item expired first, yes

Great! Thanks

> I’d be impressed if your manager agrees.

I am his manager and I agree with him

I assumed that’d be the case. What’s the average response and resolution time for pro customers? Difference for free customers? How’s your escalation process, if I may ask? (PagerDuty, etc). Can you speak to false positive rates? I break things and run media channels, would instaquit for plan B if there were an issue. (Don’t mess with my paper is the stem of my concern). I’ve designed and built a similar offering at a big B2B company and we spent considerable time coming up with heuristics to plug the gaping holes possible in this kind of system, hence false positive concerns. I’m also a CF partner. Thanks in advance for any insights or pointers.

I'm probably the better person to answer these questions, since I'm the one who designed the anti-abuse measures in question.

If I understand correctly, you're worrying about false positives, i.e. your site accidentally gets shut down because we incorrectly believe you're launching a DDoS. This is not a plausible outcome of the measures we've currently implemented.

As a CF partner, you should feel free to arrange a private conversation if you are still concerned.

Do CloudFlare Workers require external HTTP endpoints to have CORS headers like Javascript within webpages need?

No.

CORS is designed to protect two things:

- The user's cookies for other origins, which the browser will normally send on any request to those origins.

- Behind-the-firewall servers that might be accessible from the user's browser but not from the public internet.

Neither of these things apply to Workers: a Worker obviously has no access to the browser's cookie jar, nor does it have the ability to see behind-the-firewall services since it's not behind-the-firewall.

CORS does NOT protect against DDoS: Typically, CORS does not prevent you from sending requests; it prevents you from seeing the content of the responses. Any web site in a browser can use an <img> tag or submit an invisible <form> to cause a cross-origin request, CORS or not -- but it can't read what comes back.

So, Cloudflare Workers do not enforce CORS. We've implemented different measures specifically to mitigate DDoS.

Any private gitbucket/bitbucket integration on the card, like most CI SaaS do, gets an oauth2 token for the desire repositories, setups the notification webhooks, deploys on push (of nominated branch)?

We're looking into building tools like this, but also note that you can whip up something yourself that pushes via the Workers API. Maybe you can even implement it as a Worker... :)

Lack of tooling integration is one of the most challenging things about using services like yours in production. "Whipping something up" is not something I want my eng team doing. Having built a system where customers want a similar integration, I understand how difficult it is to provide!

I hear you, and we're working on it.

How does a cold start perform, I'm guessing with sandboxing of v8 vs a whole new container, and deserialization of a single script into a free sandbox v8 slot (perhaps they store the AST or preJITed code for even faster start? - not familar with v8 internals) it is much faster than cold starting nodejs on AWS lambda?

This is one of those questions I love answering. :)

It turns out V8 is very well-optimized for loading JavaScript quickly, since that's exactly what it needs to do when loading a web site with a user waiting.

We're seeing average cold start times under 5ms.

There are some features in V8 to make loading a script even faster.

I'd be glad to connect over these features on twitter (same username). I implemented them.

So no free tier? That’s kind of a bummer, but I guess it’s time I start paying Cloudflare for something. :)

Correct, no free tier. You can use Cloudflare's free tier for all our usual stuff (caching, acceleration, SSL, etc.) that's included and add this on for minimum $5/month.

Offtopic:

Can cloudflare be used to serve purely assets? i.e say you are disqus and can cloudlfare be used to serve the javascript which sites need to include in each page?

Edit: The question was misunderstood. I asked because cloudflare ToS seems to be against serving just assets and they want to serve entire sites

I don't see why not. It's just like serving the assets for your own site, except they're being pulled by other sites.

I've been doing that. The problem is when users' IP get challenged by CloudFlare the assets will not load, even with security level set to essentially off.

Asked a sales rep and got told to contact support, which is pretty much impossible as I don't know who are affected until they reported the problem to us.

Have you tried turning "Browser Identity Check" off? It's in Firewall > Web Application Firewall > Browser Integrity Check.

A few times, out of desperation, I have also tried whitelisting entire countries in Firewall > IP Firewall.

Yes. Cloudflare is, among other things, a CDN, and can do that. With Workers it becomes an exceptionally powerful and configurable CDN, as we actually expose an API to control the behavior of our caching systems to your Worker.

One too many people joked about running a Bitcoin miner... ;)

Congrats, this is really great. Question, since it’s not based on Node.js there is no way to leverage on a regular package manager, how can I create reusable code between edge workers? How can I make an open source snippet without copy and paste?

The short answer is webpack!

The long answer is webpack, rollup, or browserify :)

For now, I recommend thinking of Workers like you do browsers: Use Webpack or Browserify or one of the many other tools to transpile npm packages into a single script, then upload that to Cloudflare using the API.

Meanwhile now that the base product is out we will be spending more time improving the tooling, and integrating with Cloudflare Apps so that you can package (and sell) Workers for other people to use on their own sites.

Are the server edge-points true to UTC Time? As in, are they synced with something like NTP? There is a severe lack of accurate time coming from edge-points in systems such as AWS Lambda, and no one appears to be confirming if they do or don't make sure their time is correct.

It would be incredibly useful if there was an edge-point service that could return linux epoch time in milliseconds (Thats accuraet to withing a 1ms of UTC time). I've been working on a Live broadcasting syncing system and there really wouldn't be anyone in a better position than CDN's with lambda-like functionality.

This isn't my department, but I'd assume our servers have accurate time. But I'd also assume Amazon has accurate time, and you say they don't, so maybe I don't know anything.

Then again being accurate to 1ms worldwide sounds like one of those problems that runs up against the speed of light? We don't have atomic clocks like Google does...

Atomic clocks would get you far below the 1ms threshold. I'll rephrase to within 25ms, since that can be pretty acceptable and doable across the internet by just popping up NTP.

Without a synchronizing service, time can't be trusted to be accurate. Time should always (try) be absolute to a global time, and not individual clocks a devop manually put some numbers in for.

Right, I just find it shocking that AWS wouldn't be running NTP. But I'm not a sysadmin, I dunno.

Correction, it may not be under Amazon API service. Apologies.

Will every request to a worker-enabled site, pass through the worker v8 engine and charged at the going rate, even requests for static ressources like favicons, or jpgs etc? Or is there some way to limit the worker engine request matching to a specific area of your site(like /service)?

You can specify URL patterns where the worker should be enabled or disabled, under "routes" in the config UI. You won't be billed for requests to routes where the worker is disabled.

I wish routing and page rules had rudimentary regex support like VCL or most other location matching route libraries/servers.

Yeah, trouble is, it's depressingly easy to create a regex that performs very poorly, so if we allowed that, people could easily break the service.

However! One of the neat things about Workers is that you totally can use regexes in JavaScript, and our sandbox prevents runaway CPU usage. So if you really wanted regex-based page rules before, you can probably get that with a Worker.

We've exposed an API for controlling Cloudflare features from a Worker, including many things commonly controlled via page rules:

https://developers.cloudflare.com/workers/reference/cloudfla...

We'll be adding more in the future.

> a regex that performs very poorly

Would that still be true if you limited the regex support to DFAs instead of NFAs? I can’t think of a single use-case for backreferences in routing patterns. :)

Good idea; a nit on terminology:

NFAs and DFAs have equivalent expressive power[1]. For this application, you probably want a determinized DFA (one without epsilon transitions); that's constant time on the input string. (It's still not as fast as Boyer-Moore.)

Backrefs let you write PDAs, which are more powerful than either NFAs or DFAs.

PCRE2[2] implements DFA-based matching. The cited page says it's slower than the NFA algorithm (although presumably a lower complexity order); I think because of lookaheads and capturing.

[1] Michael Sipser, _Introduction to the Theory of Computation_.

[2] http://www.pcre.org/current/doc/html/pcre2matching.html

> determinized DFA

Yep, that's the thing I was thinking of. Thanks!

Question, since you seem like you might know: is there an alternative regex library—or a plain Yacc parsing grammar that you could stick as a validator in front of PCRE's parser—that would allow through only the regexes which parse out to constant-time (determinized) DFAs?

Not an expert in this, but I'm thinking this is usually handled by switching to a non-backtracking regexp engine. Otherwise, any such validator would depend on details of the regexp engine's implementation.

Would it be possible to invoke native code like in other FaaS runtimes: https://github.com/mankash/nativeGcpFunction?files=1

Once we enable WebAssembly, you'll be able to use compiled languages like C/C++/Rust/Go/etc. through that. It's not exactly "native code" but should accomplish the same goals.

We don't plan to allow true native-code because sandboxing native binaries is much more expensive than sandboxing with V8 / WebAssembly. In fact, for these kinds of massively-multi-tenant scenarios, code running in a V8 sandbox can actually be faster than "native code" due to reduced context switching. See: https://blog.acolyer.org/2017/08/22/javascript-for-extending...

Any way to increase the global state that can be stored and shared across instances? Say to 100mb? Could this be an option to be requested?

Tests ive run using CFW as a memory cache have been very very fast.

That limit is actually already over 100mb (128 to be exact, but there's a bit of overhead).

I can't promise we can raise it, but please feel free to reach out to us (workers-developer-help@cloudflare.com) and we'll be happy to weigh in.

Did you consider having a 5$ annual minimum charge? Or is there any deeper reason for having a 5$ monthly charge other than just a 5$ minimum charge total.

To be honest, pricing and billing is not my department. But, I don't think we have annual billing in general right now, so doing it for just one product would be weird. There's definitely a lot of discussion going on about better ways to do pricing and billing in the future.

Is WASM support in the cards? This would be awesome for isymtope.org but the engine is in Rust.

I think you just launched the next serverless resolution ;)

> Is WASM support in the cards?

Heck yeah. We almost get it for free with V8, except that the default WebAssembly API is designed around dynamically loading code at runtime, whereas we want to make sure that all code we run is uploaded strictly through the config UI/API and is available for forensics. So basically we just need to build a way to upload WebAssembly blobs and import them.

Is the issue with a length limit on data: uri's? something with the CSP?

Will there be some kind of session storage or is it completly stateless?

> Is the issue with a length limit on data: uri's? something with the CSP?

The problem is if we left the WebAssembly API enabled as-is, then people could fetch code from the internet and execute it. We already disable JS eval() so we had to disable the WebAssembly API too.

> Will there be some kind of session storage or is it completly stateless?

We're working on storage. It's a complicated problem. :) (See elsewhere in this thread.)

In the meantime, for many use cases you can use the HTTP cache as a sort of storage. Also, each worker instance has global variables which are writable -- there's no guarantee we won't reset your worker at any time, but generally one worker instance will handle multiple requests so can do some in-memory caching in globals.

This site loads very slowly, most images are in the megabyte range :(

Working on that, thanks for the feedback.

Thanks, you inspired me to take a look at Google Pagespeed as well as shrink the images further.

Would there be any issues running something like expressjs on it?

Workers are based on the Service Worker API, not based on Node.js APIs. So, Express is not likely to work out-of-the-box; it would need to be rewritten to be based on Service Workers.

The good news is that the Service Worker API is more modern than Node, e.g. using Promises instead of callbacks, which IMO makes it a better experience to work with.

Is the price for each request or is it request * workers

Same thing. Your site has only one worker script, and each request is therefore processed by at most one Worker. You can of course merge many independent pieces of logic into one script -- it's code, after all.

Enterprise customers are allowed to have multiple scripts mapped to different URL routes. This is mainly so that different teams owning different parts of the site don't step on each other. However, generally if you stuff your logic into one script rather than multiple, it will perform better, since that one script is more likely to be "hot" when needed.

ohhh, its one worker per domain, you cant have different ones that match on url? so you have to do that matching in your worker to run different code?

Correct.

Note that Enterprise customers can have multiple scripts-per-domain, allowing you to run specify a specific script per route (or routes). Additional matching logic (e.g. headers, cookies, response codes) can then be done within the Worker itself.

Hopefully that limitation can be relaxed over time. Having to stuff all my logic in one big script sounds a bit annoying. At minimum having access to a second worker script route would be welcome for testing/development purposes, so one doesn't muck up a working production script.

> Having to stuff all my logic in one big script sounds a bit annoying.

Keep in mind that you can write your code as a bunch of modules and then use a tool like webpack to bundle it into one script, before you upload it to Cloudflare.

> for testing/development purposes

I agree with this, we definitely plan to add better ways to test script changes. Currently there is the online preview that shows next to the code editor, but it's true that there are some limits to what you can test with it.

Would this be capable of doing image resizing?

I guess it would be depend on the resizing profile, fetching from a remote origin and reading into a buffer should involved <1ms of computer time (latency not an issue given 15seconds of async wait) but then guaranteeing all done in 50ms of compute time for resizing I would be doubtful, if you could get compute bound resizing node.js code to yield control the event loop every 1ms you could abort after 14ms and issue a subrequest to your own image resize processing server or something like imgix.

>all done in 50ms of compute time for resizing I would be doubtful

I don’t know the details of your environment or timing precision, but I’m quite sure that a single core on a modern server CPU can resize and JPEG compress more than 20 reasonably-sized images per second. :-)

You're probably right, presumably the 50ms of compute time, is that of reasonable 1.6ghz xeon+, and compute time is accounted as actual real compute time - not inclusive of waiting on some virtual CPU to be multiplexed onto a real core.

I benchmarked various programs to resize images a year ago. There was a JS package with native bindings to GD, a go program, and a pure JS implementation. Both go and GD were an order of magnitude faster at scaling down and converting PNG to JPG.

As the person who's desk is next to @kentonv, I think the only reasonable thing to do is to build in image resizing service, make it incredibly popular, and drive him crazy with your never ending resource requirements.

Joking aside, we don't know yet what is and isn't possible. Please build this and other things. If you need help or more resources, including CPU time, please reach out (workers-developer-help@cloudflare.com).

Happy cake day! Congrats on the big ship!

Reply on news.ycombinator.com