2bluesc

Interesting walk through in the linked video as he tried to troubleshoot this:

https://www.youtube.com/live/1UnoBfw6soI

Pretty shocking to see (extremely unlikely) non-malicious code work / not-work depending on a security mitigation setting.

Curious to see where this goes as to whether it's a kernel bug and nobody is paying attention to `mitigations=off` now or the unlikely outcome that it's an actual hardware bug where mitigations work around it and nobody has noticed.

Seems he had overclocked and maybe disabling the migation has revealed an instability in his setup that's otherwise stable (or at best marginal).

I delayed upgrading to flux and finally bit the bullet this summer, and a month later read the announcement deprecating it.

Next time around I'm going to give TimescaleDB a look.

Used to use HaGeZi's Blocklist but kept hitting false positives like this.

Also `dua`[0] is a great `du` replacement which is must faster on modern NVMe drives. It includes an interactive mode `dua i` which I'd frame as a `ncdu` replacement.

[0] https://github.com/Byron/dua-cli

This is op. Check the link.

It's mostly (all?) static content.

This should be cheap and easy to host behind a CDN like Cloudflare.

I use the `OnFailure` property to trigger a service that emails me for failed services like backups which are run as system timers + service.

I also use `failure-monitor` which is Python service that monitors `journald`.

Files on Github for those interested:

https://github.com/kylemanna/systemd-utils

Found this interesting:

> Residents voted to opt out of the state restriction 122 out of 123 times, with the lone defeat coming in Longmont in 2009 after an industry lobbying campaign.

Wish they'd offer dedicated servers in their Ashburn, VA (or Hillsboro, OR) datacenter but those seem focused on their cloud offering.

Roughly 40% (and rising) of Google users use IPv6.

https://www.google.com/intl/en/ipv6/statistics.html

I searched and couldn't find anything in the tailscale client repo. Link to the issue?

Did find headscale docs about "Setting custom DNS records"[0]. It seems only `A` and `AAAA` records are supported. This might be the start of setting up headscale this weekend.

[0] https://github.com/juanfont/headscale/blob/main/docs/dns-rec...

I'd never heard of Technitium, but was intrigued looking at. Was thinking "hmmm what could I do with this" and then had to refrain from creating another project just because.

TBH I find Docker networking a struggle and usually disable the `iptables` stuff and end up configuring my own rules. Painful, but at least less intrusive.

On the note of Tailscale+Docker networking, gluetun[0] is pretty awesome. It runs a Wireguard (not tailscale compatible, yet) instance within a Docker container and then you share that networking namespace with the other containers effectively confining them to the VPN. Comes with basic container namespace firewall configuration and DNS over TLS configuration.

[0] https://github.com/qdm12/gluetun

I know this can be done manually (and I do), but the issue with that is that: 1. It's manual 2. Single point of failure of this server that was needed

My point was that MagicDNS is implemented in the Tailscale client on each machine (fault tolerant, 0ms latency) and has almost all the things necessary (DNS resolver, push mechanism for record updates) except for a custom defined zone.

Running `drill @100.100.100.100 <node_name>.<magic_dns_domain>.ts.net` is 0ms because it's local, and doesn't depend on a single DNS server running somewhere on my Tailscale network.

For servers sure, but things like `tailscale` exist to save every laptop and cell phone from looking like a devops project.

Furthermore you could extend this argument almost every other cloud service with a primary feature of "convenience" and/or "management". Just build everything yourself.

The missing feature from Tailscale for me is the ability to host a Tailscale only DNS zone.

They have Magic DNS, but that only works for individual Tailscale nodes. I want multiple DNA records pointing to a single Tailscale node. Would be even better if I could use my own domain (subdomain even better) instead of their long `foo-bar.ts.net` domain.

Currently need to do this manually, but seems overly redundant since Tailscale already does 90% of this with MagicDNS and is fast because it's in their client vs a remote server.

Setting up a few p2p wg VPNs is manageable.

However, when you have 10 nodes and need to add one more node, you now need to update all other nodes so they can speak p2p. Management with scale is the struggle.

Had similar feelings and did like it more then I thought I could.

My escape hatch from the monopoly is headscale[0] which I can self host.

[0] https://github.com/juanfont/headscale

There's a lot of repeated stuff for no reason. The congestion control and loss recovery sections are a struggle to read.

Ctrl-f "uses a combination of congestion control and loss recovery to recover from packet loss" or "This is similar to"

Should note this only applies to AUR packages and `-git` mostly because of missing archive hashes.

AUR packages pinned to mutable tags are easiest to hijack.

This is something I've thought about. We don't really have the resources and I'd rather pay Gitlab to manage this on a day to day.

If we did self host, it couldn't be Gitlab as I fear they'd pull the plug on these offerings as Atlassian is doing[0]. The incentives are there for them to offer self hosting in the short term to grow the customer base, but long term incentives push them to maximize profit via over priced managed services.

And this is the hard part: moving to another solution and self hosting is a ton of work.

Watching the other replies to see what self hosted solutions are most intriguing.

One thing I'm doing now is halting further development of moving things to Gitlab CI/CD so we can better control our future fate.

[0] https://www.atlassian.com/blog/announcements/journey-to-clou...

Ironically, the IPv4 address for gitlab.com in my region is on a blocklist this morning[0]:

    $ drill -Q @1.1.1.1 gitlab.com
    172.65.251.78

    $ drill -Q @8.8.8.8 gitlab.com
    172.65.251.78

    $ curl -s https://iplists.firehol.org/files/firehol_level3.netset | grep 172.65.251.78
    172.65.251.78

[0] https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6...

> Yes, they are adding features nobody asked for like crazy, just to be able to check more boxes in some feature matrix.

This.

If people want "DevOpsSec" then make it a feature or addon tier. I'm certainly in favor of more security and testing, but the few times I've reviewed this feature it doesn't work for our use case. We use external CI/CD services because Gitlab CI (and runners to some degree) don't do what we want. But now, we're forced to pay for this feature as if it helps us. Perhaps Gitlab can't imagine a company that doesn't build cloud connected software.

I moved our organization of hundreds of users from Bitbucket years ago after a week long internal API rate limit in Bitbucket's infrastructure crushed our CI/CD system and took them way too long to acknowledge and fix (of course it wasn't documented). I fear this is the build-up to a similar move again, and I dread it. At the time Gitlab was on a much more impressive trajectory then Github and hence motivated my decision, but this has changed and it's largely self inflicted wound for Gitlab.

Part that pains me the most is I championed the move to Gitlab and encouraged all the teams to double down on it over the years.

This is on my todo list after wrestling docker almost in to compliance with nftables. Even so, I still have some issue with ports forwarded to docker services not NATting correctly and instead show up with a source IP of the docker bridge. Switching to nftables exclusively (docker is using iptables-nft) and preventing docker from doing this should resolve my issues.

I can't think of another application on any of my systems that muck with firewall rules behind the scenes like this.

Did you use or try Zen Ledger?

If so, they had a breach and have been very quiet about it too [0]

[0] https://news.ycombinator.com/item?id=34543078

This is the link from the post to their vendor with more details: https://www.klaviyo.com/blog/august-2022-security-incident

It appears this happened on August 3, over 5 months ago!

> ## Patch

> Since the vulnerable operation in nft_payload_copy_vlan should account for the encapsulated VLAN tag, I suspect that the last plus sign should have been a minus since it prevents any wrapping.

> ## Mitigating the bug

> If you are unable to patch this bug, disabling unprivileged user namespaces will prevent exploitation:

    sysctl -w kernel.unprivileged_userns_clone = 0

Sounds more like extortion

> Twitter or Elon Musk if you are reading this you are already risking a GDPR fine over 5.4m breach imaging the fine of 400m users breach source Your best option to avoid paying $276 million USD in GDPR breach fines like facebook did (due to 533m users being scraped) is to buy this data exclusively,

Is there a site that published this you can share?

I suppose this is how such things start.