Usually, when refineries flare something like that it's because what they are burning is not suitable for use, and making it suitable would cost more than the product would sell for.

Often methane as a by-product of oil production is flared, because the amount is small enough that it's not worth setting up processing plants and supply chains for. Other times, the fluid is heavily contaminated by e.g. sulfur compounds, and would be costly to purify. Still other times the production of the fluid is unreliable or intermittent, and cannot sustain a continuous production process.

Although, flare gas recovery systems exist nowadays to make use of these waste gases, commonly for local power production for the refinery itself.