231 karma · joined May 2, 2010
> In our codebase, we have hundreds of these jobs, each with their own run_config. If you look at this codebase as a database, you can imagine these jobs being rows in a jobs table, where the fields on each row correspond to the parameters provided to the RunConfig class.
> Wouldn’t it be great to be able to browse all these jobs in a table?
I know the article suggests a database / table, and that could work but I feel like there are a suite of things you'd want on top of that that aren't given for free -- JSON validation, history, libraries with caching, namespaces, etc.
For this exact subproblem, I created www.config.ly .
K:V would be: testing_data: { exposure: 0.10, colors: ['green', 'red'], }
// on the client:
function getColor() {
configly.get('testing_data, function(data) { const roll = Math.random(); const index = roll > data['exposure'] ? 0 : 1; return data['colors'][index]; });
}
re: global K:V. We plan on introducing teams that would work exactly that way!
Thanks!
All of the values can be fetched across projects / clients, it's essentially a simple K:V store with broad libraries. For example, if you have a key "price" whose value is a number "100", you can call:
configly.get(price)
from all your clients (provided they are all using the same API KEY).
Today, clients download at the granularity of an entire key-value. So if you changed that price to "150", it'd fetch the entirety of "150" and not just the digit flip. Were you envisioning a case of storing a lot of data?
Today, our free service is capped at 100kb total. Was there a use case you had in mind?
I view Config.ly as being useful for _anything you'd put in a class variable_. For example text copy or styling. And if you put it in Config.ly instead of hardcoding, you can avoid deploying which can be slow (e.g. for iOS it's days) -- and potentially even avoid having developers do the work altogether.
Does that make sense or do you view it differently?
> Databases can do this but often aren’t used that way for good reasons (they can be cumbersome to wire-up, there are scale concerns about adding extra load to your DB, it’s risky to touch a production database, etc).
I can think of other benefits; you generally don't want really anyone touching your prod DB directly -- but _especially_ non engineers. So what if they want to update copy? You could build or leverage a web UI... but now you're going down the path of building Config.ly. You also don't get version history for free with most databases. And once a value is in the database, you also need to worry about the middle layer of getting that value to your clients and having your clients fetch it intelligently/caching, etc.
If we build a Python / Django library, would you integrate Configly? If not, what would it take?
I think you hit the nail on the head for a core use-case: deployment processes inhibiting quick data changes. I believe the problem gets worse as you work at larger places... I've worked at [unnamed SF Tech Company with ~1k engineers] where deploying took ~a day simply waiting for the CI along with a backlog of commits from other developers.
I really like the challenges you call out. We don't do a good job of saying this on the site but we explicitly don't want h manage secrets / sensitive data; especially because the API_KEY is on the client libraries - so there isn't really anything stopping anyone from peeking in memory / the source making requests (I think this is different than say a more traditional login-auth system where the secret is stored in a session and only the person with access to the client with that session can access that data).
100% agreed on latency and availability. Config.ly needs be super solid here (sacrificing of course some consistency... slightly delayed updates). Besides following best practices for building/designing/operating high-avail systems (both my cofounder and I have experience here), there are some clever things we can do on the client. For instance, caching should help with both and I thought some of commenters below mentioned had great suggestions for other sorts of clever client fallback mechanisms we can do to help here [1] [2].
Thanks again for the great input!
[1] https://news.ycombinator.com/item?id=25062496 [2] https://news.ycombinator.com/item?id=25063062
You could think of this as Redis hosting (configured to be durable!) + UI with features like version history, lightweight type checking (and potential for user accounts, schema enforcement) + client libraries on a variety of languages for super easy install / integration with all of your frontends. The libraries are smart with things like caching + a team focused on improving this feature set / functionality.
>I think if battery/data even becomes a concern, you have already put too much data into config and probably need a database? > Does anyone really care about a couple hundred KB (at most) these days? Or am I underestimating the config size/usage?
I could be wrong but I think this is a concern some mobile developers have. It came from user feedback but we should definitely look into it more.
> Who has access to each client's database? Is it audited? Is it encrypted at rest? I'm sure it is, but Config.ly would be wise to add this information to avoid fears.
This is great feedback, thank you.
- Updating content is way simpler. Even non technical folks can do it. (and soon you'll have revision history)
- I think with the S3 library you can use HTTP caching; Config.ly can do smarter caching (e.g. if you fetch (a,b) and then just a, HTTP caching would fail).
I could be wrong, but I think some folks would want their other clients -- even their server -- retrieving that value from that one single source. So when you roll an Android app, you'd also want to implement HTTP + caching... And basically you're on the path to building your own Configly.
That's something we can definitely give more priority to.
I actually had an iOS library version that would poll infrequently in a background thread and had a synchronous API. Something like:
Configly.init(API_KEY, { values: [keyOne, keyTwo]}) // (other code runs) print(configly.shared().get('keyOne')) // would be cached by the background thread
but was slightly concerned about unwanted bandwidth/battery usage for mobile users but perhaps that mode would be useful exactly for the situation you describe.
Situations with a slow deployment process has been a big motivator for building Config.ly -- and I've seen it happen at companies with medium-sized eng teams (taking hours to swap values during incident fires), and I know it's a problem _all_ iOS developers have due to the app store deploy process.
I also think there's other value here... For instance having a UI means non-technical folks can update it. A lot of my time as an engineer has been spent making copy tweaks (I've felt existing CMS products were too heavy-weight and difficult to plugin).
Also, having turn-key libraries mean you don't have to copy/paste hardcoded values among many clients (or store on the server build an API for _every_ constant). I'm a big fan of DRY.