We use cookies on our websites. Information about cookies and how you can object to the use of cookies at any time or end their use can be found in our privacy policy.

Opinion 5 min read 6 comments

DxOMark isn't as reliable as you think

As soon as a smartphone is released, mobile tech lovers turn to DxOMark. What will be the score of the new top-range device? Will the iPhone beat the Pixel this time? Will the triple cameras on the Huawei P20 Pro even make a difference? Cameras seem to be getting increasingly complex, boasting more sensors and advanced technologies, but ones that are reaching high marks are not always the best. Let's take a look at what's going on here then.

Whenever a new top-range phone is released, such as an Apple, Samsung or Google, we see it fly past other competitors in the DxO list. Just like that, a device that was ranking towards the top will have to settle for fifth or sixth place. Does the device take bad pictures because the brand isn't keeping up with the times? Take the Mi Mix 2S for example, why doesn't it deliver more than the iPhone 8 Plus?

First let's find out what DxOMark is, and why it dictates the market. This is how they define what they do:

The DxOMark is the industry's trustworthy standard for camera and lens image quality measurements and ratings. For years we have been recognized for providing the most rigorous hardware testing using industry-grade laboratory tools in our analyzes to establish the most comprehensive reference database with our thousands of camera test results and lenses.

In each test, they evaluate elements such as exposure, contrast, dynamic range, color saturation, hue, white balance, color shading, texture, noise, auto focus, artifacts (including frame softness, distortion, vignetting, chromatic aberration, touch, reflexes, ghosts, aliasing, moiré patterns, flash, zoom at varying distances, bokeh effect (otherwise known as portrait mode) and these are also tested for video recording, too. You can learn more about their process below:

All this is done with thousands of tests with photos. As you saw in the video, they have special labs where they take the same pictures in the same light situations, but also go outside to do new tests. Until 2017, the test didn't take some things into consideration. Now, new items started to integrate smartphone cameras, so the evaluation has changed somewhat since last year.

It's not without problems

But, the question remains: why are there devices with questionable quality in the ranking? One of the problems lies in the latest changes, and another in the weight that the evaluator gives to new developments in features like zoom and bokeh effect. In many cases, good cameras that don't have these items, end up losing to others that do not have the same quality, but can offer these extras that don't always add value for the individual user.

AndroidPIT huawei p20 pro 2994
Huawei's P20 Pro is currently the highest-rated device on DxOMark / © AndroidPIT by Irina Efremova

Many other producers and content analysts I've spoken to have come up with an important point: manufacturers have figured out how to score well in DxOMark. Looking at other tests, it's not too difficult to figure out which aspects are the focus of the company's attention, like zoom and bokeh as I mentioned, as well as contrast and sharpness.

Important factors such as wide lenses, black and white photos and RAW format are not considered in the running, although they are important for the photographic quality. Because these important aspects don't carry that much weight in the scoring, good cameras are unable to score highly, and manufacturers are paying attention only to the factors that are rated, and could even be using software that DxO sells itself to analyze cameras.

The third problem is that DxOMark is not an NGO: it's not just a camera analysis company. The main activity of the company is consulting for companies that work with cameras, so the tests and ranking are nothing more than an advertisement for their own primary service, which is to advise hardware companies to improve their cameras. Do companies that receive advice from DxO get better places in the ranking?

It's impossible for a company not to have its biases, but these factors are a little worrisome. Combined with the fact that there are no other large companies carrying out technical tests like DxO, we have a problem. Many consumers unfortunately rely on the ranking to decide what to buy. The company does, however, try to put our minds at rest by stating that it does independent testing, and wouldn't tarnish its reputation so willingly.

In the end, all that matters is your taste and your budget

There are many benefits to using the DxO website, and it certainly knows what consumers are looking for, so it is useful to check it out before making a purchase. Always take the score with a pinch of salt though, and trust your own judgement with what you consider to be decent photographs, as well as making sure you look at reviews and photo tests before making your final decision. Take into account what you consider to be best without leaving the final decision to the ranking. Zooming to you may not be as important, but having RAW could be for example.

Ideally, you should always have as much information as possible, and then build your own opinions based on your needs and how much cash you have to splash on a new smartphone.

Do you trust the DxOMark ranking? Tell us in the comments!

53 Shares

6 comments

Write new comment:
All changes will be saved. No drafts are saved when editing

  • I wont trust totally but I will consider their score before I buy


  • Zed 2 months ago Link to comment

    Awesome fact, like this article!


  • i think benchmarks are ruining the phone camera business. my s9 takes horrible photos. sure, they are sharp and detailed most of the time, but it can't handle contrast at all. any bright light blows out a whole section of the scene. yet, DXO gave it a super high rating? my s8 was better. just waiting for google cam port to fix this for me. they seem to be the only ones that can get the software right. shame.


  • I know that the phone in the top photo, it's a honor 5x


  • No ranking, no benchmark, no opinion is free from imperfection and personal tendency. Not even robots are faithful, provided that they are created, built and fed with data by humans with different views and interests. That's why, most of the societies recognize that the exclusive responsibility belongs to the ones who sell, and some other, also to the merchandise that spread lies or mediocre products at the price of gold.
    I always choose the products I buy, according to my risks, taste and purse. The best cars I owned and drove never deserved 4 ou 5 stars in 5 given by "specialists". For me, the important was that they valued 6 stars out of 5.


    • As far as reputation goes, it's all we have right now. Sure problems are there, but no one can tell me a better standard. It's hard to please everybody, no camera can produce the perfect photo, there's just too many people to satisfy their views. Accuracy can be a personal taste for different people.