Well, Japan is the
only mature industrial nation other than the United States where a majority of science papers are published in domestic journals[1][2], many of which are predominately or solely Japanese language[3], whereas international journals--including international journals in Japan--are invariably English.
Of course the Japanese scientists you've met speak English. People who don't speak English well enough are not generally going to participate in events or situations where they would need to rely on English-language speaking skills. That someone apologies for their bad English is, of course, irrelevant; if it were truly bad you wouldn't have encountered them. Likewise, Japan has many international science programs, forums, and conferences; and they'll be filled with proficient Japanese English speakers, but Japan is a huge nation so it says little necessarily about broader patterns.
I don't doubt that a majority of professional research scientists in Japan have some proficiency in English. Especially the top scientists; even in Japan "impact" would undoubtedly be heavily biased toward publication in prestigious international journals. But there's plenty of literature on the internet that describes the relatively poor English-language proficiency (especially conversationally or with verbiage outside that in their technical field) and poor English-speaking research community integration of the broader Japanese scientific establishment. See, e.g., "Japanese materials scientists' experiences with English for research publication purposes".
I first encountered this cultural dynamic when researching the tilting of the Millennium Tower. While friction piles weren't first used in Japan, Japan was where friction piles were first heavily researched and employed for large buildings, including skyscrapers. People kept saying friction piles were idiotic in earthquake-prone, sandy soils, but that's precisely the environment where they were first researched and from whence the global engineering community's confidence sprouted. However, AFAICT much of that research was published in Japanese, even through the late 20th century and (IIRC) even early 21st century. Was that Nobel-worthy science? No, but it was very useful applied science; and applied science which was highly consequential to global engineering and architecture, notwithstanding that much of the substance of it existed on the other side of a Japanese language barrier.
[1] See https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8024886/
[2] But see, e.g., China and Brazil where domestic publication dominates and even outpacing international publication: https://www.digital-science.com/blog/2018/11/japan-collabora...
[3] E.g. many of the publications by the The Japan Institute of Metals and Materials are only or predominately in Japanese. Some domestic publications require summaries and tables in English but permit the text to be in Japanese; e.g. the Journal of the Society of Materials Science.