Reading this makes my stomach sore.
Remember that RGB is not a colour space. Neither is CMYK. Colour encoding models do not necessarily imply a colour space as outlined by the definition of an RGB colour space via ISO 22028-1.
Even Lab encodings will vary based upon the assumed white point.
What is not expressed in saying "RGB" or "CMYK" are the other critical facets that define what they are referencing. In RGB, what are the colours of the three lights and what is the combined white point? In CMYK, what are the colours of the inks, the paper colour / type, and the illuminant in question? The mixture of the unsaid contexts are what define how colours result to a standard observer.
In the case of Lab, one must take care to assert that the same formulation has taken place before comparing values. For example, it isn't enough to just say "RGB" because of the facets above. What are the colours of primary lights in the source RGB space? The destination you are comparing? What are the white points? What is the white point assumed in the formulation for the Lab conversion?
Only when all of the contexts are identical do the values take on meaning, and a valid comparison within the limits of the encoding model is possible.
TL;DR:
* Make sure you are aware of the primaries of the RGB spaces being compared with Lab.
* Make sure your white points are aligned on the RGB spaces being compared with Lab.
* Make sure your RGB spaces are aligned to the white point of the Lab formulation in question.
* Make sure you are linearising your RGB spaces correctly before converting to Lab; every RGB space typically has a unique transfer function.
* Be well aware of the limitations of Lab, such as red turns yellow and blue turns purple hue nonlinearity etc.
* Be using an appropriate model for the intensities of light being compared. Lab, for example, is not well suited for anything beyond diffuse white reflectance values.
Colour should be able to handle all of the above points, and deliver great looking plots as well!
You sound like a perfect candidate to dig into Colour. There are a great many extremely knowledgeable people around that project that can help.
Mark Fairchild's Color Appearance Models is probably about the most canonical tome one can recommend, with the caveat above.
To do what you are seeking, you would need an accurate profile of your camera in question, as well as likely a spectrophotometer for the objects in question. It is a pretty tricky field.
I would encourage you to contact the Colour Science Twitter feed to get in on the Slack channel for help and guidance.
See you there!