PDP-10 was an architecture, and there were many operating systems running on them. You may be thinking of TOPS-20, which indeed used filename.ext, where the extension was limited to 3 characters.
There was also ITS, which was one of the famous operating systems on the PDP-10. On ITS, filenames consisted of two parts, rach being 6 characters separated with a space. So you could have a file named "abc def". The second half was often used in the same way we use extensions today, but executable files were named "ts name", where name was the name of the command.
I'm not sure what format filenames had on TENEX and TOPS-10.