I tried the same thing with GPT-4 set to Temperature 0
===
System Prompt: You are an assistant who specializes in the Mesoamerican calendar.
User: Is the 1-earthquake inscription on monument 3 at San Jose Magote is the oldest known instance of a Mesoamerican calendar date?
Assistant: Yes, the 1-Earthquake inscription on Monument 3 at San José Mogote is considered the oldest known instance of a Mesoamerican calendar date. This inscription dates back to 600 BCE and is associated with the Zapotec civilization. The monument itself is a carved stone slab that displays a combination of the Mesoamerican calendar's 260-day ritual cycle and the 365-day solar cycle, which together form the 52-year Calendar Round.
User: What does 1-earthquake mean?
Assistant: In the Mesoamerican calendar, 1-Earthquake refers to a specific day within the 260-day ritual cycle known as the Tzolk'in. The Tzolk'in consists of 20 day signs and 13 numbers that are combined to create unique day names. In this case, "1" is the number and "Earthquake" (also known as Caban in Mayan or Ollin in Nahuatl) is the day sign.
1-Earthquake (1-Caban or 1-Ollin) represents the first day of the Earthquake/Caban/Ollin day sign within the Tzolk'in cycle. Each day sign has a specific meaning and symbolism, and the Earthquake/Caban/Ollin day sign is associated with movement, change, and the natural forces of the Earth. It is believed to symbolize the dynamic energy of the Earth and the transformative power of natural events.
User: What is the previous 1-earthquake to 7.16.3.2.13?
Assistant: To find the previous 1-Earthquake date before 7.16.3.2.13 in the Mesoamerican Long Count calendar, we need to calculate the number of days between the two 1-Earthquake dates.
The Tzolk'in calendar has a 260-day cycle, and the Earthquake day sign repeats every 260 days. The Long Count calendar date 7.16.3.2.13 corresponds to a 1-Earthquake day in the Tzolk'in calendar. To find the previous 1-Earthquake, we need to subtract 260 days from the given date.
7.16.3.2.13 in the Long Count calendar is equivalent to 1,244,813 days from the calendar's starting point (0.0.0.0.0). Subtracting 260 days gives us 1,244,553 days. Converting this back to the Long Count format, we get:
1,244,553 days = 7.16.2.4.13
So, the previous 1-Earthquake date before 7.16.3.2.13 is 7.16.2.4.13 in the Mesoamerican Long Count calendar.
===
I studied history in undergrad and still keep up but I know nothing about Mesoamerican timekeeping so I don't know how accurate this is, but it seems much more internally consistent than ChatGPT (which is probably GPT-3.5-turbo with whatever temperature settings they use?) LLMs are bound to be bad when you get to such specialized knowledge which its training sets have little of. It was like when I went to grad school I stopped being able to Google for theories, formulae, and lemmas in my classes because most of the knowledge I was learning was only found in papers. Once you're in a specialized field of study, no model can help. But not many people are working in those fields, and those folks should understand that their knowledge resides only with them.
I don't think this is a good example to show how people learn incorrect things from LLMs. LLMs come with the same caveat as Google or Wikipedia in that, the rarer/more specialized the knowledge, the more original research the person needs to do to corroborate their answers.