there went 10 mins of debugging why kafka wouldn't start
207 karma · joined July 25, 2009
We're excited to test our Clickhouse integration with Clickstack, as we believe OTel and Clickhouse make for a powerful observability stack. Our open-source Rust OpenTelemetry collector is designed for high-performance, resource-constrained environments. We'd love for you to check it out!
We’ve been thinking along similar lines with the use of Rust, particularly for OpenTelemetry collection in environments where high performance and low resource overhead are critical, such as edge and serverless. With that in mind, we’ve open-sourced a lightweight OpenTelemetry collector written in Rust to address these use cases. We’ve also developed a native Lambda extension around it, and have seen encouraging interest from folks aiming to improve cold start times.
The project is still fairly early, but we’re optimistic that Rust can open up new opportunities for efficient observability pipelines. Vendors like Datadog are also moving in this direction with their Lambda extension and appear to be adopting Rust more broadly for data-plane components.
If this resonates, feel free to take a look here: https://github.com/streamfold/rotel. We’d love to hear your thoughts on how this could be useful.
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (Real Time Logging), AppOptics (Server, Infrastructure, Application Performance Monitoring and Distributed Tracing), Pingdom (DEM) and Loggly (Structured Log Analysis).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, ClickHouse for large scale log storage, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and 50TB of logs per day.
All team members, whether in one of our offices or those remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
At SolarWinds Cloud you get all the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
Learn more at: https://solarwinds.jobs/jobs/?q=cloud or contact me directly at mike-at-solarwinds.cloud (no recruiters).
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (Real Time Logging), AppOptics (Server, Infrastructure, Application Performance Monitoring and Distributed Tracing), Pingdom (DEM) and Loggly (Structured Log Analysis).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, ClickHouse for large scale log storage, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and tens of terabytes of logs per day.
All team members, whether in San Francisco, one of many offices, or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
At SolarWinds Cloud you get all the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://solarwinds.jobs/jobs/?q=cloud or contact me directly at mike-at-solarwinds.cloud (no recruiters).
* Impact on M5/C5 instances over similar time period, any difference with the Nitro hypervisor?
* Were Dedicated instances (https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...) patched as well?
* Other examples of software that adapted batching performance automatically with increase in call latency.
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (hosted logs), Librato (time-series metrics) and TraceView (APM and distributed tracing).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and tens of terabytes of logs per day.
All team members, whether local in San Francisco or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
Papertrail/Librato/TraceView are wholly owned by SolarWinds Inc. so you get the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://www.librato.com/jobs or contact me directly at mike@librato.com (no recruiters).
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (hosted logs), Librato (time-series metrics) and TraceView (APM and distributed tracing).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and tens of terabytes of logs per day.
All team members, whether local in San Francisco or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
Papertrail/Librato/TraceView are wholly owned by SolarWinds Inc. so you get the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://www.librato.com/jobs or contact me directly at mike@librato.com (no recruiters).
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (hosted logs), Librato (time-series metrics) and TraceView (APM and distributed tracing).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and tens of terabytes of logs per day.
All team members, whether local in San Francisco or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
Papertrail/Librato/TraceView are wholly owned by SolarWinds Inc. so you get the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://www.librato.com/jobs or contact me directly at mike@librato.com (no recruiters).
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (hosted logs), Librato (time-series metrics) and TraceView (APM and distributed tracing).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and tens of terabtyes of logs per day.
All team members, whether local in San Francisco or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
Papertrail/Librato/TraceView are wholly owned by SolarWinds Inc. so you get the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://www.librato.com/jobs or contact me directly at mike@librato.com (no recruiters).
We're looking for a full-time software engineer to take a key role in building the large-scale distributed systems that power Solarwinds Cloud products: Papertrail (hosted logs), Librato (time-series metrics) and TraceView (APM and distributed tracing).
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second and tens of terabytes of logs per day.
All team members, whether local in San Francisco or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
Papertrail/Librato/TraceView are wholly owned by SolarWinds Inc. so you get the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://www.librato.com/jobs or contact me directly at mike@librato.com (no recruiters).
Librato, located in San Francisco, develops a unique cloud-based platform designed to help companies collect, visualize, store and create alerts on large amounts of streaming data to manage the health of their applications.
We are currently hiring Data Engineers who want to help the build systems that power the backend infrastructure of Librato. We are looking for someone that enjoys working on large-scale distributed systems problems to build a metrics and monitoring solution used by thousands of customers.
We’re a small team so everyone has the opportunity to have a big impact. We’ve built our platform out largely on Java8 Dropwizard services, a handful of Golang services and some C++ where performance is critical. We leverage Kafka as our main service bus, Cassandra for long term storage, our in-house stream processing framework for online analytics, and we rely on Zookeeper as a core part of intra/inter-service coordination. Our data pipeline pushes millions of messages a second.
All team members at Librato, whether local or remote, commit code to Github, communicate over Slack and Hangouts, push code to production via our ChatOps bot, and run all production applications on AWS. We also use an array of best-breed SaaS applications to get code to production quickly and reliably. We are a team that is committed to a healthy work/life balance.
Librato is wholly owned by SolarWinds Inc. so you get the benefits of a small startup, with the backing of a big company so there is no worry about the next round of funding. SolarWinds offers competitive bonus and matching 401k programs that create an attractive total compensation package.
If this sounds remotely interesting I’d be happy to spend a few minutes talking about what you are looking for and whether what we are doing would be appealing.
This is an example of some of the technology we build and work with on a regular basis: http://www.heavybit.com/library/blog/streamlining-distribute....
Learn more at: https://www.librato.com/jobs.
Apply at: http://solarwinds.jobs/san-francisco-ca/sr-data-engineer/AE5...
Librato, located in San Francisco, develops a unique cloud-based platform designed to help companies collect, visualize, store and create alerts on large amounts of streaming data to manage the health of their applications. Our parent company SolarWinds provides powerful and affordable IT management software for the enterprise to more than 100,000 customers worldwide. SWI has $500M in revenue and 40% growth year over year. Together we offer application performance management for any environment, with unexpected simplicity.
We are looking to hire full time software engineers for our data and operations teams. We are passionate about monitoring, run entirely in AWS, develop in Java/Python/Golang/Ruby/C++, and believe in a healthy work/life balance.
This is an example of some of the software we build and run: http://www.heavybit.com/library/blog/streamlining-distribute...
More info at http://librato.jobs
Librato, located in San Francisco, develops a unique cloud-based platform designed to help companies collect, visualize, store and create alerts on large amounts of streaming data to manage the health of their applications. Our parent company SolarWinds provides powerful and affordable IT management software for the enterprise to more than 100,000 customers worldwide. SWI has $500M in revenue and 40% growth year over year. Together we offer application performance management for any environment, with unexpected simplicity.
We are looking to hire full time software engineers for our data and operations teams. We are passionate about monitoring, run entirely in AWS, develop in Java/Python/Golang/Ruby/C++, and believe in a healthy work/life balance.
This is an example of some of the software we build and run: http://www.heavybit.com/library/blog/streamlining-distribute...
Data Engineer Position: http://solarwinds.jobs/san-francisco-ca/senior-data-engineer...
Operations Engineer Position: http://solarwinds.jobs/san-francisco-ca/operations-engineer/...
More positions at http://librato.jobs
So this is definitely not a dig at any framework. If we were starting to build stream processing into our infrastructure today we would most likely start on an existing framework. Also, I imagine as our needs grow we will leverage these frameworks in future proof-of-concept streaming projects that may or may not make sense to move to Superchief.
Superchief is not an off-the-shelf-framework that anyone will be able to drop-in to fulfill their stream processing requirements. We were running on Storm for 2.5 years prior to moving to SC and during that time we truly developed the understanding of the system and our own workload to be able to build SC. SC builds on many assumptions and understandings of our workload that we did not have when we started with Storm.
For us we decided this was the path that would lead to the quickest and highest reward, and we were able to justify the effort by not only halving our infrastructure footprint, but being able to scale the next 10x. Other companies, eg. Twitter Heron, have made similar realizations.
So what was their response? Was their response the `ec2-authorize` command to run?
# cat /etc/yum.repos.d/adobe-linux-x86_64.repo
[adobe-linux-x86_64]
name=Adobe Systems Incorporated
baseurl=http://linuxdownload.adobe.com/linux/x86_64/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux
# yum install flash-plugin