Every day at the same time, my internet dies for 1 minute. How do I investigate? | Hacker News Reader

Every day at the same time, my internet dies for 1 minute. How do I investigate?

308 pointsoverallorder136 comments

3:40pm every day my wifi loses internet access. My devices remain connected to the network, but all traffic dies. Almost exactly 1 minute later everything is resumed. I have no idea what the cause could be.

How could I begin investigating this? I have a spare Raspberry Pi and and old Android phone at my disposal, and some programming competency.

View on HN

Your ISP does that. Apart from IP rotation theory, I heard that some ISP billing software requires a session reset to flush network statistics to a billing database, because it is accumulated in RAM. They have to do that to block/divert clients whose account reaches a limit, because you don’t normally upload payment info to your firewall/proxy and only check it for new sessions. (I’m not an ISP guy and this may simply be an urban legend from 90's)

My experience partially confirms that. I had an ISP who would redirect every syn packet on 80/443 to internal “low money” site in the night, but I had to reset my router after sending them money, cause otherwise that crippled session got stuck at their hardware until next autoreset.

1 minute may be a technical pppoe cleanup/reconnect timeout. Some shitty ISPs reset this timeout even for unsuccesful attempts, and this deadlock can last for half an hour until e.g. a router decides to back off for a while. Could be cured by turning it off for 3-5 minutes to cool off the ISP side. It’s rare these days.

> 1 minute may be a technical pppoe cleanup/reconnect timeout.

i'd second this guess because its a common setup. OP could check the external ip before/after the reconnection and verify this way if it changed.

would also be helpful to know connection technology (dsl/cable/wifi/fibre..) and name of the isp

If you want to confirm this, if you can pull the logs from your router you should be able to see this happen and getting log entries around that time.

1. Unplug your Router at some other time, for at least 5 minutes, long enough for your ISP to notice it's absence and give you a different IP address. If your 3:40 PM changes, it's the 24 hour expiration of your external IP.

2. Do a ping 8.8.8.8 from your Linux machine (Raspberry Pi) or ping -t 8.8.8.8 from windows, and watch what happens at 3:40 PM

3. As others have said, turn off uPnP

This assumes that the ISP has a short lived DHCP lease. Comcast Business once tried to charge me to roll my DHCP IP from them. They told me I would have to disconnect my equipment for three weeks for the lease to expire. I told them to go fuck themselves, got a minicipal fiber connection, and never looked back.

Must be nice! I wonder if OP even has the ability to switch ISPs (here in Texas, limited options)

With Comcast, if I spoof my router's MAC address, I get a new IP instantly... otherwise it is basically static (residential service).

So does that mean they somehow detect the mac address has come from a new device and roll the ip address to prevent people from doing server kinds of things?

I host servers on my home connection and never had issues (but I do use a dynamic dns service to make sure that I always have the correct IP)

Consumer ISPs don't always refresh IP addresses like that. I have had the same IP address on my home internet for years and was told that I would need to purchase a block of static IPs in order to recieve a new one.

A) There are a lot of potential hiccups with trying to get a new IP:

- Some ISPs assign statically (odd, but true) meaning you will always get the same public IP

- The DHCP lease time is different for many ISPs: 5min to weeks, meaning you’d have to be offline for at least that long in order to be assigned a new IP when connecting

- Some ISPs lease it to the modem and not the router or anything internal (does not apply to what you said, but another tactic: changing the MAC address)

B) OP asked how to investigate. If getting a new IP dodges the issue and it doesn’t happen again, then there’s basically zero chance that the actual cause will be known

Launch ´mtr google.com’ on one of your devices (preferably connected via Ethernet to your router) a few minutes before the cut off, and watch what happens when the cut occurs. It will show you exactly which equipment on the route starts dropping.

Although I suspect your ISP has a 24h lease and your modem renegociates at 3:40 everyday

So then the trick is to forgo internet by disconnecting power from a few minutes before that time to the middle of the night and get power back? I do feel for the ISP's with this setup because the right thing to do would be to write in the installation manual: "Only install in the deep of night".

Well they could 1) renew the IP before it expires or 2) reboot the modem remotely at 3:30am for example.

Or assign a lease that expires at 0330 instead of a lease that expires in 24 hours.

But I’ve not experienced similar in 20 years of dynamic ips.

it depends. depending on the dhcp implementation on the server side you may have the same problem (ie while you get a new lease it may expire at the same time). unlikely but I’ve seen some stuff when it comes to dhcp

Actually it seems that it is not renegotiating everyday: it is expiring and re-requesting.

The way DHCP 'should' work is that when a client is part-way through the lease it should try to renew the IP it already has:

* https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Pro...

* https://tools.ietf.org/html/rfc2131#section-4.4.5

This allows the client to (hopefully) keep extending the IP it has so that a re-IP does not cause current connections to drop. If the modem is not doing this (can the OP log into it in someway to see logs?), then it is acting as a 'non-ideal' DHCP client.

Until this is sorted out, try rebooting the modem an an 'odd' hour that will not disrupt you during the day. This is no guarantee though: the DHCP server may remember the old/current DHCP lease and simply re-issue it with the same expiration time.

I'm on DSL/PPPoE, so this may not apply, but: my Asus router has a setting that allows it to automatically reboot. I do this at ~04h00 to get a new IP every day so help with privacy concerns. I generally surf with cookies disabled, so these two things help with the low-hanging fruit of simple tracking techniques. (My DSL modem is bridged.)

I had this exact issue in 2008 in SF on Comcast Business.

The lease would be dropped and renegotiated every few hours, we could notice it because our SSH tunnels would get torn down.

What was annoying was that the business plan had a static IP associated with it, presumably managed by their DHCP setup.

It took weeks to convince Comcast the issue wasn’t in our building, and moments for them to fix once they “got it”.

What piece of evidence or logs finally convinced Comcast that the issue was on their side?

Honestly my memory is not that good! I remember a few tech visits as they replaced modems, etc, and I eventually got to some “3rd level support person” that said something akin to “oh, that issue again”.

Did you try saying shibboleet?

https://xkcd.com/806/

c.f. Judges 12:5-6 (https://www.bible.com/bible/114/JDG.12.5-6.NKJV):

> The Gileadites seized the fords of the Jordan before the Ephraimites arrived. And when any Ephraimite who escaped said, “Let me cross over,” the men of Gilead would say to him, “Are you an Ephraimite?” If he said, “No,” then they would say to him, “Then say, ‘Shibboleth’!” And he would say, “Sibboleth,” for he could not pronounce it right. Then they would take him and kill him at the fords of the Jordan. There fell at that time forty-two thousand Ephraimites.

Haha - This is and the comment above are awesome. But not really a backdoor.

I saw this daily drop on several small biz Comcast setups. I recall I found it while looking through all the settings on the router, though I do recall having to call Comcast and they set something so it never happened again. It was the Comcast device with built in wifi. Hardwired didn't glitch, just wifi.

I could be wrong but this sounds more like NAT port issue given the frequency and static lease agreement on IP.

This is the correct answer. Although typically leases will attempt a renewal halfway through the lease time, so it would be a 48 hour lease.

As OP said, mtr will tell you where in the line it’s happening.

Here's a summary from the man pages of mtr, for those who are not familiar with the tool but maybe are reading HN on their phones so can't check it out:

    mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.
    
    As mtr starts, it investigates the network connection between the host mtr runs on and HOSTNAME by sending packets with purposely low TTLs. It continues to send packets with low TTL, noting the response time of the intervening routers. This allows mtr to print the response percentage and response times of the internet route to HOSTNAME. A sudden increase in packet loss or response time is often an indication of a bad (or simply overloaded) link.
    
    The results are usually reported as round-trip-response times in milliseconds and the percentage of packetloss.

This actually makes me wonder about whether there's a command that'll let me see short summaries of what every file under /usr/bin is, in the form of a list. Definitely wasn't aware of mtr as an average web developer up until now.

apropos -s 1 . will list a short description from the manpages (might have to run mandb first)

On the subject of Lease Time.

My router is currently showing a lease time of 3 Min. I think most router tends to renew their IP at 50% of ISP's lease time ( in this case 6 min )

What are the purpose of these ridiculously low lease time? I remember in the old days they tend to go for 48 hours if not longer.

To get you to pay more money for a static IP.

Seems Reasonable, but I though most ISP dont want Home users to be using Static IPs.

And apology for asking an off topic question. I guess that is what the downvotes are coming from.

There's a limited number of IPs so most homes CANT use static... we already don't have enough IPs for the mountains of devices.

But the other part is money... why give something away for free when you can make it a "feature" to pay for?

Tons of other reasons in various articles...

https://www.online-tech-tips.com/computer-tips/ott-explains-...

https://superuser.com/questions/590391/why-do-isps-change-yo...

I work at an ISP and setup both static and dhcp services. Statics take far more work and management than dhcp, ignoring the scarcity problem. We make you pay more because you are requesting something that requires both documentation and management on a home connection.

Id be glad to give them away for free but it is all downside for us and over 99% of residential customers dont want one to start with.

I think we have 3 residential customers out of 1500 that have statics.

After having read some of the articles on it (including the two listed), the mechanics are something that are probably a lot more intensive to manage than I figured initially. Changing DHCP is a couple changes and then... wait. Changing hard coded paths and all that...

I'm actually glad I don't deal with that stuff on a day to day... but I like learning about it.

un-ignoring the scarcity problem, when I've gotten a static IP I would usually get a block of 8 and they would waste the 0 for network and 7 for broadcast. I've wondered why they didn't just give you 8 usable addresses.

(or if I set up my own router could I just use the all?)

Most ISPs are using carrier grade NAT (CGNAT), so most users are sharing IPs with other customers.

Is there a citation for the "most" here? I understand this is true in some cases, but I've never used an ISP that does carrier grade NAT. I'm on Spectrum cable now, which is not exactly known for being a good ISP, and I have my own address.

MY INTERNET HAS BEEN GOING OUT AT 3:40pm PT TOO!

Sorry for the all caps. I just scrolled through my texts and saw the “internet out?” texts are at the same time.

Hopefully that means it’s a provider issue behind us. Wonder what we have in common?

- South Lake Tahoe, CA

- Spectrum Gig internet

- Asus Zenwifi mesh

Hmmm my Spectrum Gig in OR doesn’t go out at 3:40p but it does pretty much every day at 1pm. I was similarly excited to see this post...

In my neighborhood it’s shortly after midnight. Or so I’ve heard from my kids and neighbors. I’m never awake to notice.

Also Spectrum, but no gigabit here.

Spectrum in Huntington Beach, CA; ours does go out for one minute during the day as well though I didn't think to get systematic and note the times.

My spectrum cuts out pretty reliably at 12:30am every night. Lets me know its time for bed.

Now that, I could get behind: “Attention Spectrum user, cease doomscrolling at once and get some sleep”

Do you have xfinity by any chance? And are you using their x1 combination modem/router in bridge mode with custom (non-comcast) DNS settings?

Because the exact same thing was happening to me at the exact same time. I’m no networking expert but when I checked the logs it seemed that every day Comcast would try to “fix” my (not broken) DNS settings. I had been using mullvad DNS and open DNS as backup but since switching to nextDNS the issue has stopped.

I don’t know enough about networking or DNS to offer any explanation as to why.

I have xfinity configured with custom DNS and used to have a somewhat similar problem. At the same time each day, DNS would stop working (but I could still reach plain IPs) for a couple minutes.

It went away after I called them and had them "de-provision" and "re-provision" my modem. Basically deleting it from their system and readding it. For whatever reason, whenever I'm having chronic comcast weirdness, having them do this solves it. It sucks that this is the way things are.

Are you familiar with the "Reply all" podcast? They have a "recurring segment, called 'Super Tech Support' [in] which [...] the Reply All team [...] takes on odd or especially complex tech support issues that the listeners or friends of the hosts have encountered." Not sure if your situation qualifies, though.

[1] https://en.wikipedia.org/wiki/Reply_All_(podcast)

I had a similar problem once. The issue ended up being a router power cycle which occurred at the same time every single day. It took several days to diagnose. First, during an outage I tried to login to my router. Just because the internet was down the local area network should have been up. I couldn’t access the admin panel for several minutes. Next, I started observing the lighting patterns on the router. I came to realize that it appeared to be restating itself for no good reason. In the end my internet provider replaced the router which fixed the problem. Tech support was pretty stumped. A few people on the internet complained about the same issue with the Fios router model I had (don’t remember which one it was). Good luck! Let us know what the problem was if you figure it out.

I had a similar problem with my Asus router. Traced it back to it being scanned for UPnP every day at the same time. Something about the scan caused the router to crash and reboot. Disabled UPnP and the problem disappeared.

This is a known "feature" of some access points (AP), especially those used on big corporate WAN's. They want to keep the hardware costs low and software development/testing costs low, so instead of making sure the AP is absolutely solid in operation, they reboot it once a day to clear any problems. Even if the AP has dropped off the network it reboots at a preprogrammed time and gets back on the network. Also saves sending someone out to press a reset button. It also gets rid of any attack code that may have taken over via RAM but which is not present in the firmware PROM. I worked on a campus where all our wireless AP's would reboot every night at around 3am. Data collection for experiments in labs would either have to have retries built in, or use the wired network. This is not an uncommon commercial tactic because it takes an enormous amount of testing to try every possible situation a router or AP could get into (malformed packets, unknown bugs in standardized protocols, compiler bugs, overheating when a sunbeam shines on the router, ...) You might be able to change the time this reboot happens in the admin settings of your router to be something more convenient for yourself. That could also ensure you that as to the cause of the restart.

Did you check if your public IP changes after this minute? I don't know if this still happens but back in the past my router did this every 24h to get a new IP.

I can confirm this. The time is around 7:00PM and the IP address changes, and I have to log in again to some websites because my IP has changed. Downtime is approximately 2 minutes.

This was my thought as well. It sounds like a 24 hour DHCP lease expiring.

A lot of service providers do this because they do not want you to host static services from your "dynamic" home IP, as they are selling "business" lines with dedicated static IPs. Or at least they used to do this for that reason. By now it's just some thing a lot of ISP do. I have seen other possible explanations like ISP claiming this is a "security" feature because attackers cannot have permanent access without always learning the new IP, which is supposedly somehow hard. This is pretty bull tho. If attackers have access to your system they could just run software signalling them back any IP changes. And even nmapping the entire IP space of an ISP to find a service you were hitting again is very feasible.

Anyway, if the ISP does this, then you have to reconnect once every day (or sometimes every two days), and if you don't do so manually they will just cut your connection on their side and make you reconnect.

My (German) ISP does that too, the one I had before did it, the one before did it as well.

If you can replace the ISP provided router with your own you should be able to renew the lease.

At 3:39p, unplug your router and leave it unplugged for an hour, then plug it back in. If the outage then moves to 4:40p every day, it's a 24 hour IP address lease expiring.

Are you talking about IP leases from the router to the local network or IP leases from the ISP to the router?

likely router -> ISP - if it was the devices, they could show up as not being connected to the network. also weird that all devices' leases would expire at the same time

If your devices are kept connected this means that your wifi and router are still active, probably the issue somewhere behind your router, either a connection or provider. But due to exact timing it's likely something on the provider's side and not just some connection issue. I would personally start with contacting them to investigate.

You might also check the router's logs for that time, perhaps there's something useful information.

Some german ISPs used to force a reset of home internet connections every 24h to trigger IP changes so that static IPs would remain reserved for the business tier. The time of the reset could be shifted by preemptively reconnecting at a desired time, they only reset the link after 24h of actual uptime.

I get why they do it, but actually causing issues just to cycle IPs seems like a lot. I've had cases where I have the same IP assigned to me for quite a long time.

They already did this on dialup and ISDN, back when routers didn't exist. When DSL first started appearing they just kept doing it. The idea was to not keep unused connections/IPs up, since you didn't have a router you would need to reconnect manually.

Also, it made it harder to run servers until we had services like DynDNS.

On newer connections it isn't common anymore, especially if you have a voice service.

On my family's FritzBox we can set the time for that connection reset. I think it's somewhere around 04:00.

Probably not your issue: Internet: Old TV caused village broadband outages for 18 months https://www.bbc.com/news/uk-wales-54239180

I was thinking of this too.

Disconnect your wifi gateway and plug a computer directly in to your modem. Eliminate or prove the router as the problem source.

Sounds like standard reconnect every 24h, many ISPs do that. reset your modem at a weird time to get the interruption at that weird time in the future.

This.

I think the OP is so eager to get back online he sees 3:40 every day and is using that timepoint as his troubleshooting clue.

However If he were to disconnect 3:10 and then just be offline an hour until plugging the device back he would probably see the reset happen 4:10 going forward.

@OP did you try calling support to report the problem? It's almost impossible to diagnose ANY internet problems (not related to your own equipment of course) and the ISP has tools and access to do it. In any case, it's much easier and faster to contact support.

In my experience, even if I discover what is the problem, if it's on their side I still have to go through support to get it fixed, so I just wasted my time. And in most cases the lowest level of support doesn't even write down the information I give in the ticket, even if I send an email.

1. Start the ping command at 3:39pm and leave it running until 3:41pm then make a note of whether packet losses start at 3:40pm. (E.g. "ping yahoo.com" on Unix or "ping -t yahoo.com" if on Windows).

2. If there are packet losses, then you have your answer: the issue is from your ISP.

3. If there are no packet losses, then you'll need to look closer at your network. Check to see if some hardware on your network might be performing a reboot at the 3:40pm mark.

Reminds me of this news article: Old TV caused village broadband outages for 18 months https://www.bbc.co.uk/news/uk-wales-54239180

Oy! You got a license for that broadband jammer mate?

"My internet" is a bit vague - so I am making some assumptions here. Scenario (A) could be that your WiFi dies at a specific point in time. You might then want to look at some source of electromagnetic interference (like a motor starting while creating sparks). Scenario (B) could be that your internet service provider is cutting your connection and provides a new IP address - which is common for many xDSL providers. You want to check if the external IP address of your router changes. You may also find some log files in your router that may give you more information. Some routers allow you to select the time when the router preemptively disconnects at a certain time in order to avoid the disconnect from the ISP at an arbitrary moment. You may want change this setting or to talk to your ISP and try to convince him to not cut the connection during the day but rather during the night. Scenario (C) could be a device on your local network creating interference. Your tools of choice will be tcpdump and WireShark to locate the culprit on your LAN. Enjoy your debugging and good luck!

You don't need something fancy like motor sparks to kill a 2.4Ghz WiFi, a leaky Microwave will do that and perfectly explain the 1 minute length.

Switching to 5Ghz will help if this is the case

I would check out DSLReports and ask them there. http://www.dslreports.com/forums/all

Chances are they experienced it there too. But I had an issue like that with Cablevision at 3-4am when they would push out firmware updates to the modem.

Have you tried different WiFi channels?

I had a similar experience to this years ago while working at my mother's kitchen table for while. The network would keep going dead and I had no idea what was causing it.

Eventually, I discovered that it happened every time she used the microwave to heat up a drink or something.

Changing to a different WiFi channel fixed it. :)

I came to suggest the microwave too. The 24h IP reset sounds more likely, but don't discount the microwave, they typically use the 2.4 GHz frequency (as do bluetooth and some wifi networks of course). So, if the 24h IP reset isn't it, try switching to a 5 GHz wifi network to see if the same dropout occurs. If that solves it, it could be a nearby microwave oven.

Step 1: reboot all devices on the network: computers, router, cable modem (or whatever equivalent device your router plugs into).

If that doesn't fix it, below is how I'd proceed.

Use the trace route networking tool (google it, different syntax on different OS) to some known good public IP address. I usually use 1.1.1.1 or 8.8.8.8.

This tool shows you the path your traffic takes:

    $ traceroute 1.1.1.1
    traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
    1  192.168.1.1 (192.168.1.1)  1.286 ms  1.386 ms  1.963 ms
    2  192.168.11.1 (192.168.11.1)  2.591 ms  4.342 ms  5.219 ms
    ...snip...
    16  0.ae19.GW8.CHI13.ALTER.NET (140.222.230.223)  42.888 ms  35.742 ms 0.ae20.GW8.CHI13.ALTER.NET (140.222.230.225)  41.060 ms
    17  152.179.105.202 (152.179.105.202)  51.913 ms  50.681 ms  55.442 ms
    18  one.one.one.one (1.1.1.1)  70.087 ms  65.130 ms  53.607 ms

The snippet above shows that I have two local devices (192.168.1.1, 192.168.11.1) my traffic passes through and 18 devices total before my traffic hits the public IP.

Run this command at 3:39. Then run it again at 3:40. Then again at 3:41. Compare results and that should give you a good idea which device is malfunctioning at that time.

Some routers also have a ping command baked into them. If you can make it constantly ping 8.8.8.8, while you run the test above, it may help you. In the case where your traceroute stops at your own router, you can look at the ping from the router itself, and see if it also stopped:

A) If the traceroute stops at your router, but the router ping is not interrupted, then something in your router is flaking out.

B) If the router ping is also interrupted, then the problem is "upstream" from you. Maybe the cable modem (or whatever device) that is upstream from your router.

If you decide the problem is upstream, contact your ISP. If it's local, investigate and/or replace the flaky device.

HTH.

Not likely your issue but I had something similar that exactly every day my Wi-Fi would act up and my computer disconnect. It also happened to be the same time a nearby church starts ringing it's bells and my room happened to be very close to it. It also didn't happen on days the church didn't use the bells.

Still don't know what exactly caused this and I don't live there anymore but because I was the only one in my family with these problems, and only my room was facing the church, I want to believe the bells rang at a frequency that messed with my old routers Wi-Fi.

If someone could tell me if this Is possible, I could finally get some closure on this.

A lot of church bells these days are not real bells, they are PA systems playing a recording of church bells. They are often programmable and wirelessly controlled, for example:

https://www.brgproducts.com/carillon_pricing/index.php

So it seems entirely possible the wireless part of the church bell system was interfering with your router. If the church did have real bells it seems very unlikely they would generate strong EM radiation at 2.4GHz or 5GHz. They might induce some vibration if they are really loud, which could trigger an intermittent connection in the router, though that seems pretty far fetched also.

Hack at the problem until its reduced to its purest form.

> My devices remain connected to the network, but all traffic dies.

Can the devices send packets within the network, but not the internet? Use ping to double check. if no, do you have any wired devices to confirm rather or not its wireless interference vs the router doing some sort of maintenance task.

Does this impact all protocols? tcp/udp/imcp. I've seen random network hiccups only impact tcp before.

ping tests imcp, dns (nslookup on windows/host on linux) tests udp (and can also test tcp) http tests udp and tcp, depending on browser and service. curl can let you confirm the test is going over tcp.

MTR is likely your final solution for investigating this. It is like a traceroute that rapid fires out to get second by second details about all the hops in a network path. The types of errors you get can also tell you why.

If it ends up being tcp only and internet based, things get harder, some mtr clients support using tcp instead of imcp to find hops where tcp breaks, but your isp routers might also refused to reply to such packets directly.

Finally, watching the lights on the router and modem (if seperate) can be illuminating. First, get a feel for normal light operation, normal activity blink speed, etc, then starting a few minutes before the event normally happens, just observe the lights for changes until the event ends while also using your phone to know when the event has started and ended.

On that note, look up how to access web consoles on the router (and modem if seperate). They may have event logs that tell you if anything is happening, like ip renewals, or if they are getting commands from the isp to do things.

What type of connection are you on? Is it DSL or fibre. If it's copper then my guess is that there is a device (like a heating pump) that's on a timer with a big switch thats creating a pulse of radio noise that's then echoing up and down the copper and pushing the session out of bounds. At that point the far router (in the dsl box on the road) will try and resync the connection.

First step check if the external wan IP address has changed.

Second Check the logs of the pppoe daemon in your router. (If the router provides such logs).

I'm actually surprised by all of the comments. The community jumped right into solving OP's problem and has largely ignored the OP's original question of "how could I begin investigating this?" Another variation of this question could be "how do I troubleshoot complex network issues?"

After 20 years of network/system/software engineering across a wide variety of network sizes and levels of complexity, I teach people the following method to troubleshoot complex network issues:

1. Write down the symptoms that you're seeing. How do you know something is wrong? What do you see happening?

2. Draw a diagram that includes all of the components and nodes involved. (this is hard)

3. Develop some hypotheses that could be worth testing. (this is hard and highly variable based on knowledge/experience)

4. Establish a "test plan" that allows you to prove/disprove the hypotheses while making minimal changes to the system. Start from one source device and work your way out to the farthest component you identified in your diagram. Start at the lowest OSI layer and work your way up.

5. Methodically test components, step by step, along the diagram.

6. As you test, record your results. Add new hypothesis to the list as you gather more information but don't start testing them right away! You can develop a new test plan after you finish the first one that proves/disproves your new hypotheses.

7. Repeat all steps, adding new information, until you find a solution.

I know this seems like a lot of steps when you're just troubleshooting a WiFi issue at home--it may be overkill. However, this framework applies to that scenario or when you're diagnosing any complex network issue. You'll learn more about the systems, protocols, and devices that truly make up the Internet than you could ever imagine. Through repetition and experience, these steps will get easier and some of them can happen quickly and in your head.

To help you get started, here's some info for steps 1, 2, and 3:

1. From your post: "3:40pm every day my wifi loses internet access. My devices remain connected to the network, but all traffic dies. Almost exactly 1 minute later everything is resumed"

2. Start your diagram off as an "equipment string diagram". This will include all physical devices between you and the Internet. Along the way, you may need to modify the equipment string diagram to include "virtual" devices, network segments, various protocols/servers, etc. Your diagram should include at least the following items:

  - Your laptop/desktop.  If it's happening to all devices, then pick one.  Identify your IP and MAC addresses
  - The WiFi access point that device is connected to.  Identify the IP and MAC addresses
  - Any switches, firewalls, modems, etc that connect from your WiFi access point to your Internet Service Provider (ISP) and their IP and MAC addresses.
  - The "next hop" from your modem into the ISP's network.  You can use a cloud to represent the ISP's network that you don't know/understand, but always identify the IP address of the device that your modem first reaches.  If you can find the MAC address (or other OSI Layer 2 address) as well, even better.

That's the equipment string between you and "The Internet". For now, you can ignore the complexity inside the ISP's network and beyond. You might have to add more of that later, but start small.

We know there are other components involved in making the Internet work that introduce complexity and can cause issues along the way. Let's list them on your diagram and identify what servers are used and where they might be located.

  - DHCP (local to your network and also between your modem and ISP)
  - DNS (could be local to your network and often is a third party service either run by your ISP or not)
  - Encryption (VPNs, SSL certificates, network device clock settings, etc)
  - IP routing (what devices do IP routing?  Hint: all devices that operate at Layer 3, using IP, do IP routing--including your workstations)

3. Some hypotheses (some were identified in the comments):

  - Is there a device in the equipment string that is rebooting every day?
  - Is DNS intermittently failing?
  - Is DHCP releasing/renewing your IP address assignment?  This could be from your device -> local DHCP server OR your modem -> ISP DHCP server
  - Is there an upstream connectivity issue with your ISP?
  - Is your WiFi access point losing connectivity?

The Universal Troubleshooting Process has been a web site since the mid-90s:

http://troubleshooters.com/tuni.htm

Thanks for sharing!! I've never seen this before.

I will point out to all the people surprised at the time lining up that if the issue is a timer then this is likely very common. There are only 1440 minutes in a day, and waking hours are half that brinignrh it very close to the birthday paradox (double). https://en.m.wikipedia.org/wiki/Birthday_problem.

If you consider 4 time zones in the us it's actually more common than birthdays since you'd divide by four again and people would suspect both a company wide tx synced issue and a per tx issue.

With 23 people about 1/3rd of the people commenting at this point we get a chance of 2 people colliding as:

Full day: 16% 12 Hour: 29% Across time zones: 79%

Make sure all your network devices / computers are synced via NTP, or at least accurate to the second.

Hardwire the Pi to the network. Start a verbose traceroute and ping loop on a device on WiFi, as well as the Pi. Once it fails, observe the lights on your router/wifi device(s). Note time to the second of the start and finish of the outage, and the behavior of the lights. Check the results of your traceroutes/pings. Start digging through the logs of each network device.

Also, you can never quite rule out power, even if the lights stay on. A power conditioning UPS is useful here.

Just how I'd approach it but I'd plug my machine (wired) into the closest egress point possible (cable modem, DSL modem router, whatever), open up the Network pane, run a reliable ping, load up the modem/router's status page, then see exactly what happens at that time.

This would let me pick up on disconnects, DHCP lease issues, and a variety of other things in one hit. And if it wasn't any of those, would allow me to explore some other possibilities such as the problem being at the ISP's end.

Sounds very regular, may be worth contacting your ISP to see if that is their maintainence period or any other regular operation they may perform.

I am with a broadband provider in the UK called Zen, at 23:15 every Sunday our PPPoE session is terminated, and it takes a couple of minutes before it successfully resumes. Occasionally it will happen mid-week, however still at 23:15. At one point I suspected the provided FRTIZBox router, however the problem persisted once we moved to an MT992 modem and a Unifi USG.

I had the same problem with and older Netgear router. It turned out to be a bug in the scheduling settings. It had the ability to disable network access between specific times on a daily schedule. The feature was enabled by default, but no times were set to disconnect. You'd think that would mean the connection would never go down, but a firmware update changed that. Every morning at ~3:00am the network would go down for 1 minute. Disabling the feature entirely resolved the problem.

I had this problem on my wired connection with Windows. Exactly 24 hours after boot, I would lose my connection with certain services (mostly realtime games! very annoying!) After some investigation, I realized that my computer renewed its DHCP lease with the route at this time, and switched to a static IP allocation. Problem solved.

Honestly, this should not cause anything to lose its connection (the renewed IP address is the same), but it did, so I worked around it.

Many people have mentioned DHCP -- and a misbehaving DHCP client could certainly be the culprit.

In doing some of my own network investigating recently, I discovered that services like Suricata will (by default) take down network interfaces and restart them when they get new rules. I wonder if something software on your router is doing something similar: Are you running Suricata or Snort?

Also: Logs are your friend. If you have access, review the logs of your firewall / router / etc.

One thing is to find out if something has a midnight bug, but has its internal clock set to something other than your actual time zone. I once committed a midnight bug in a manufacturing plant, and it took me a while to diagnose, because none of the PC's were networked, and I had never bothered to set the time on them. I puzzled over it until one of the operators said: "It always happens at the same time every day during my shift."

Disclosure: I'm smarter now.

It was easy way of dealing with router hanging up due to accumulated problems used by amateur internet providers. Just disconnect it from power once a day.

You can do that with cheap socket timer.

Not sure why would anyone set it to the middle of the day. Maybe error in setting times or clock or it was set intentionally so that you can service it at normal hours if it doesn't come back on.

The problem is you can put in 'off' time and 'on' time but they have to be at least 1 minute apart.

What channels do you use on wifi? If you you use DFS channels on 5GHz it could be a radar issue that forces your router to switch channels. I have it as well, but it happens at night so I'm not terribly concerned about it (my router sends me notification thats how I know).

Edit: https://en.wikipedia.org/wiki/Dynamic_frequency_selection

The devices are staying connected, so I doubt that this could be the source of the problem.

This could also be a power or electrical interference issue. I would start by logging the connection with ping every 1 sec or so to see if it's only 3:40pm.

Are you on a xDSL service of fibre?

On xDSL services, some monitored alarms may cause line problems when they "check in". In these cases an in-line DSL filter should do the trick.

Another possibility is that some modem/routers have a "self healing feature" that is just really a restart. See if there's anything like this in the admin/system part of the configuration settings.

Similar situation. Occasionally super bad packet loss within the network. https://use-cases.org/2020/12/29/diagnosing-eero-issues-with...

Still haven't resolved it, but it is likely Fing on Big Sur.

Check this article. Might be some other faulty electrical device: https://www.ispreview.co.uk/index.php/2020/09/18-months-of-v...

I love that almost everyone here has some sort of "family IT syndrome" but still got nerdsniped but the question

Does your WiFi provide a way to monitor the real time bandwidth usage of each device? I was experiencing randomly timed outages for a while. Turned out to be our various phones and other devices randomly saturating my low upload bandwidth with cloud backups. That shouldn’t kill my downstream bandwidth, but it does.

It doesn't kill your downstream directly, but you can't send the acknowledgements for the download fast enough if the upload is saturated. That kills your download speed

The sun was hitting my cable line at the same time every day causing my internet to go out, my router to reboot and the line power to be automatically increased. I finally called out a tech who fixed some old splitters and now the strength isn’t just barely marginal enough that the sun would cause a drop.

Second possibility: Did you connect a Sonos Device to your router using ethernet? If so, does your Sonos send on the same wifi spectrum your router does? My Play:5 sent on channel 11, so did my router, mesh repeater could not send to base while all connections appeared to be just fine.

https://www.ispreview.co.uk/index.php/2020/09/18-months-of-v...

I am using LTE internet via Huawei 3372 and it stops for about a minute or two exactly 6 hours after starting the modem (of course during some important call). After it, existing connections continue to work. I'm curious to know what might be causing this.

I have the same wifi issue. At 3pm (approx) my comcast business wifi drops for a minute or two.

Some routers allow to reboot daily at a set time. Worth checking the settings of your router.

Wow I have the same problem here. In my case I found out it is at&t themselves.

It is likely to be the automatic channel selection. It is a process of selecting the best channel and normally you lose connectivity. If not that, you should be a scheduled maintenance procedure from your isp

I am not an expert whatsoever. I just notice a similar thing happening at around the same time, the wireless and sometimes ethernet connection dies. Usually a minute up to three minutes the connection resumes.

At 3:30, disconnect your router from your modem and connect a laptop directly to the modem. No wifi. Start a speed test or long video or something and just watch the modem through to 4:00.

HAVE a separate modem and router.

We had this problem in our house and it turns out it's because I accidentally plugged the modem into one of those auto power switches meant for turning off lights periodically.

I had a connection through a small municipal ISP until moving recently. My connection would drop for several seconds at exactly 23:59 UTC every day. Never was able to figure it out.

Maybe the router is using DFS channels and some radar station takes a reading at that time every day and forces your router to change channel on its 5GHz network.

Switch off you WiFi (and your modem if they're different). Stay up very late and switch them back on at 2am or so. Maybe the problem now occurs at 2am.

It's probably the 24 hour IP reset, as many commenters have said. You could try time-shifting it to a time when you're normally asleep.

Go to 192.168.0.1, log into your router/modem and see if there's an admin "Event Log"...could find some answers there.

Friendly advice, investigating this can only be worth the time it takes if you care about the learning or treat it as entertainment.

This is most probably because you have dynamic IP address and your router resets at the same time every day to fetch new IP address.

did you check your router (sys)log messages. Many of the routers will allow enabling debug logs as well.

Try and get hold of your Internet/WiFi router's logs. The answer will probably be in there.

This sounds like your DHCP lease is expiring and you have to request a new one each time it does.

somebody (like neo) is trying to get in/out of the Matrix. Your internet connection is the most suitable path for them.

Is your ISP att/fiber? I have exactly the same problem every night at 4am.

+1 on @detaro's message. Sounds like a 24h disconnect. Your router may have a feature to set dis/connect times manually ir you unplug/replug it at the time you wish to have the dis/connect to happen.

Change your DNS servers.. I use openDNS, but there are lots of other choices. If the Pi isn’t in use set it up as a Pi-hole (DNS server / ad blocker), so you’re not use your isp provided router for DNS.

DNS probably has nothing to do with disconnects, this kind of random suggestion unrelated to the problem always makes me think of the Futurama Robot Judge: https://www.youtube.com/watch?v=Jqn8HB2w2Fs (skip to 0:15 for the punchline)

Or maybe it is related, in which case I'll eat at a hat.

I've had outages that were due to my ISP's DNS service going down... one of the reasons I started using OpenDNS.

I need to watch more Futurama. That was great!

Maybe your ISP resets your external IP at that time.

Are you using Spectrum / Charter by chance?

Use traceroute to find where your packets die?

Mine dies for 10min

traceroute?

Get better hardware and software that properly reports error messages instead of hiding them.

Reply on news.ycombinator.com