What I learnt in Dubai: Cameras!

madbilly

Hi everyone,

Following on from my previous posts about what I learnt in Dubai here's a post dedicated to cameras. I'm no expert on cameras, and my memory is now a little hazy but I hope that I manage to share with you some interesting things which are news to you. (note that all my photos were taken with a Nokia 8 on latest (at the time) Android 8.1 with stock HMD camera on auto mode). @SeqTre, @nitheesh, @sajwaz, @singhnsk, @wilson shrestha, @juho lemmetyinen, @nokianchetan and @shaunmoroz , you were there too, did you learn anything else? Please also tag others who were there.

In the Q&A with Juho he compared the HMD app to the GCam:

• Daylight performance is a case of personal taste - some may prefer what GCam produces, others may not
  
• HMD don't want to over process or colour-pop
• In my opinion (not Juho's necessarily) the HMD camera produces more natural and photographically correct images than GCam does, in daylight
  
• In low light there is a clear difference however
  
• HMD consider below 100 lux as low light - for info, this is probably higher than the lighting we had in the room where we discussed with Juho, see below photo:
  
• GCam has machine learning powered processing which helps with low light; HMD don't have this
• Gcam solution is very powerful, HMD are working on making theirs comparable; Juho is not happy with the current status quo
• In the Nokia 8.1 the significant sensor improvement compensates for low light so there is not so much need for AI on this new phone
• (Personally, I'd prefer if HMD just gave us bigger lenses and better sensors and didn't spend time on AI, but that's my personal opinion)

Also in the Q&A with Juho he explained that some Android apps, e.g. Facebook, don't actually use the whole image processing pipeline! They use a shortcut. This is obviously bad for HMD publicity, but why isn't it also bad for Google and other camera vendors? I presume that if some of the pipeline is missed out that the phones with the best optics then look the best, which is where Nokia phones with ZEISS optics should do well! Shouldn't they?

At the launch event after the Q&A I spoke with a gentleman from the MENA region office and asked a few questions which he was reluctant to answer (!). One was about the PureView trademark, which is publicly known is now owned by HMD (see EUIPO website) but for some reason he only wanted to talk about the Nokia 8.1 and not future unannounced possibilities... I can't imagine why?! (it was the launch event of the 8.1, so it's understandable). However, he did explain that there was a gentleman from ZEISS elsewhere in the experience zone so that is where I went next...

madbilly

... and as others can confirm I spent a long time talking to him! Unfortunately I didn't note his name and don't want to risk getting it wrong (or spelling it wrong) but he had some great exhibits:

• Many lenses and camera modules from previous Nokia phones
• Disected! (cut down the middle) versions of the Lumia 1020 module (sacrilege!) and the 7+ module (I think; it could have been the 8 Sirocco; I'm sure it wasn't the 8 or 8.1).
• Cut-in-half SLR lens which he told me was the best SLR lens it was possible to make, costs ~€4000
• In the picture below (this one taken with flash) you can see the SLR lens, the 1020 camera module, some other modules (can't remember which these were, maybe they were the 7+ ones), an 8.1 with a close up photo of the cut-through 1020 and the "oscar" which the team at ZEISS were awarded for their work (https://www.oscars.org/governors-awards/ceremonies/84th-sci-tech-awards-memorable-moments):

• As I said I had a long chat with him, I'll try and capture the main things of interest to me - please ask any questions as it's possible we talked about it and I've not written it down here
• See the complex shape of the lenses in the Lumia 1020. This is the case (or should be) in all modern smartphone cameras. The complex shapes are to compensate for chromatic aberration, which cannot be done in software
• Spherical aberration can be compensated for in software. He also mentioned other optical defects which could be compensated in software. Chromatic aberration is the big problem which needs good lenses to compensate for
• We discussed multiple camera setups, I wanted to compare the 8, 7+/8 Sirocco and 8.1. He first explained that the 8.1 is not a multi-camera setup. The monochrome sensor is only for depth sensing. It can produce a picture (a very good one) and if you hacked the firmware you could extract it, but that's not what HMD/ZEISS use it for
• Multi-camera setups come in two different kinds: Camera arrays, where multiple identical or similar cameras work together and computational photography combines the images to create a single, better (e.g. higher resolution) image (like arrays of radio telescopes)
• The other kind of multi-camera setup is where the cameras are different, e.g. one wide-angle and one telelphoto - this is what the 7+ and 8 Sirocco use and in this case the cameras operate independently (as far as I know, if I'm wrong please correct me, I'd like to know)
• Regarding the 8, it doesn't have the best camera setup, there are limitations (I note the aperture and sensor) and it was also the first collaboration between HMD and ZEISS; everything since then has been an improvement (8 > 7+ & 8 Sirocco > 8.1) (my opinion)
• He explained smartphone cameras are always a compromise, in contrast to the SLR lens where €4000 really does get you the best possible (his words, not mine, I don't know about SLR setups).
• For a smartphone there are always restrictions, e.g. budget, space, phone thickness, processing power
• For budget he says (for example) if the camera budget was €30 then there are various options for spending that. One choice could be multiple cameras (e.g. 1 colour camera at €18 and a mono one at €12), another could be a single large main camera with a small depth sensor (e.g. €25 on the main camera, €5 on the depth sensor); neither option is certain to be the "best" as each has different strengths and weaknesses
• (This makes me wonder what strengths of the Nokia 8 dual camera setup we have not yet discovered!)
• Regarding space and phone thickness, I asked if we would be likely to see a camera like the 808 or 1020 again and he explained that this isn't possible with current smartphone design trends - in future, these trends may change, but for now it's very unlikely
• Processing power is important for multiple camera setups as the images need to be processed, compared, combined, etc The more cameras which operate together the more power is needed
• HDR photography increases the required processing power significantly, and to do this with multi-camera setups really does require dedicated processing power (see my post elsewhere about the Light camera array)
• (going back to GCam, I would add that GCam also has excellent HDR processing, probably better than HMD, which will also help with low light performance)
• I tried a few more probing questions about multi-camera setups. For example, how many cameras could be added, realistically, but he didn't really answer with specifics only explaining that it depends on budget, space and processing power
• He explained a bit about the history of ZEISS' current partnership with HMD: ZEISS only do exclusive partnerships, they think they can only really produce the best if they are working with a single partner, otherwise they would have issues with who they could share what info with, and what info/ideas came from where.
• As you know, ZEISS have a long history of collaboration and partnership with Nokia. This partnership contract was taken over by Microsoft when they purchased the Nokia phones business. Microsoft, as we know, basically gave up their work on phones and after the Lumia 950 they didn't do any more work with ZEISS. When the new Nokia phones business was started by HMD ZEISS asked Microsoft to end the contract, which they (thankfully!) agreed to and ZEISS were then free to enter a new partnership with HMD.
  

madbilly

• Since my subtle questioning wasn't working very well (!) I decided to ask more direct questions about the various rumours I'd read about the Nokia 9 penta-lens camera monster!
• First I asked about this multi-lens patent which NPU keep referring to. FYI, I always believed it was a red herring, an old patent and related to revolving lenses which is not what the Nokia 9, if it exists, will use. Thankfully, he confirmed this. He said this patent was produced in the 2000s and would have been used on a chunky, flagship feature phone (maybe smartphone, according to definitions at the time). It is too big to use on a modern smartphone
• When we were talking about camera arrays he did refer to the Light camera a few times, explaining that they have lots of cameras with different characteristics, they're not all identical. I'm not sure how this conversation relates to what I was trying to learn about the Nokia 9, perhaps not all five of those cameras are identical?
• I did ask how FIH Mobile were involved in the partnership between HMD and ZEISS and didn't really get an answer. To me this is an important point as the software for the HMD smartphone cameras seems to be the weak point and it's not clear to me where the problem is, whether it's in HMD's part or FIH's (Evenwell's) part).
• At the end of our conversation I remarked with a cheeky grin that I was looking forward to seeing what HMD and ZEISS produce next, and in reply I got a **** expression that suggested that all those rumours about the Nokia 9 might actually be true! (FYI, I also made the same comment to Juho, and got a very similar response!).

In the spirit of cameras, here's all my photos and video from the event! https://photos.app.goo.gl/CTS4z5DvpGihzVTd6

@SeqTre, @nitheesh, @sajwaz, @singhnsk, @wilson shrestha, @juho lemmetyinen, @nokianchetan and @morozshaun, you were there too, did you learn anything else? Please also tag others who were there.

Cheers 

PS - if I remember anything else I'll add in the thread below. (but three posts is already enough, right?!)

stipe1906

I love this. It's awesome, So insightful. Just like the other posts in the series )

madbilly

@stipe1906 consider it my little contribution trying to return the favour to you and the Nokiamob bunch, plus Steve Lichfield and all the others over the years who've given me lots of news and insight into Nokia phones

stipe1906

@madbilly
Hehe, no problem I admire everything you documented. You (and others, too) were like a sponge in Dubai. Great idea from HMD to bring fans. Hopefully it will continue on next events, too.  

madbilly

I totally agree, I hope that HMD can do this for each of their launch events.

wilson shrestha

@madbilly This was really insightful review. You are truly awesome for writing this in so much detail. I have read all the things that you had to say and I'm going to comment on Gcam Vs Nokia Stock Camera topic.

And, I think I have to disagree with you on this:

madbilly said:

• (Personally, I'd prefer if HMD just gave us bigger lenses and better sensors and didn't spend time on AI, but that's my personal opinion)

I feel like AI on smartphone photography is the need of the hour regardless you have a bigger lenses or better sensors in your phone. Because Google Camera is a valid proof to this. 

To my knowledge HMD is slowly making progress on their camera software with the recent updates to their camera app with the new Pie OS on different Nokia phones that lacked these feature before and they are mostly focusing on the Pro mode which is like their speciality. Pro mode is great and all but let's be honest here, its for someone who really likes to tinker with the given setting and for those who plan to spend tons of time taking a single great picture. 

Now, In real life situation currently normal people don't like to get into this mumbo jumbo of the Pro Mode, they just want to take out their smartphone and click and get the nicest picture. They don't care about the process behind this and surely don't want to spend time in it. This is where the Google Camera outshines because it appeals to the masses with its magical AI and computational photography. Even professionals praise Google Camera for its simplicity.

The state of Nokia Camera in front of Google Camera is like this specially in low light photography:

In Nokia Camera, you will take tons of time tinkering with the exposure,shutter speed etc. on the pro mode to click a nice picture in low light where no one knows how much time will it take for you to click a perfect picture.
But, in Google Camera you take out your phone and click the picture whether its low light or not, you don't have to care the rest is done by the camera software. It will surely give you a nice picture.

I will personally choose Google Camera for the reasons I listed above. 

I think Nokia should certainly invest time in AI for cameras. With, all the smartphone manufacturer gravitating towards AI, Nokia surely shouldn't be left out. We surely, can't expect Nokia Camera to perform like Google Camera at the moment but it doesn't mean that Nokia should give up on this. They should keep investing time and keep experimenting with their software.

petrus

I enjoyed most of this, and I'm excited about this article. Thank you @madbilly ; 

naufrago

Thank you @madbilly
I think you walked around with a recorder. I would never be able to come back with such an information load.

naufrago

Now, In real life situation currently normal people don't like to get into this mumbo jumbo of the Pro Mode, they just want to take out their smartphone and click and get the nicest picture. They don't care about the process behind this and surely don't want to spend time in it. This is where the Google Camera outshines because it appeals to the masses with its magical AI and computational photography. Even professionals praise Google Camera for its simplicity.

@wilson shrestha

I am 100% agree on this.

wilson shrestha

naufrago said:

Now, In real life situation currently normal people don't like to get into this mumbo jumbo of the Pro Mode, they just want to take out their smartphone and click and get the nicest picture. They don't care about the process behind this and surely don't want to spend time in it. This is where the Google Camera outshines because it appeals to the masses with its magical AI and computational photography. Even professionals praise Google Camera for its simplicity.

@wilson shrestha

I am 100% agree on this.

Thank you @naufrago. 

madbilly

@wilson shrestha I agree that people, in general, don't want to mess around with the Pro mode. Especially at a party. A party is the use case we're all talking about here isn't it? Low light, little time, just want to point and click and get a good photo.

I may be remembering with rose-tinted spectacles but I think I could do that with my N8 and Lumia 925. I'm certain it was possible with the 808 and 1020 PureViews. That's why I say I want better hardware, because those cameras were better than the current HMD cameras (maybe N8 wasn't overall better, but the Xenon flash helped a lot; and maybe the 925 depended on the PureView software to get it's results, so I agree it's not all about hardware).

I'll also be very happy to have better software! Especially software that actually works as advertised and doesn't contain bugs and broken features...

I may be misunderstanding something as well. When people talk about "AI" in photos what do they mean? Do they mean machine learning? If so, how is this used in the camera app? It could be used at the point of taking the photo, to determine the current scene and choose appropriate settings to make the end photo pleasing to the viewer. Or it could apply to post-shot processing which is done before the viewer sees the photo. In practice, I think that Google use both, whereas most other phone manufacturers have historically only used the first one. In any case, the machine learning must have been done offline in advance by Google, it's not done on the phone itself is it? What does it learn from? The photos we delete? I don't know all this though, it's only my opinion based on thinking about it.

If by "AI" people just mean computation photography, like HDR, then that's different from machine learning, although the phone's upfront "AI" in auto mode can determine how to use HDR.

So, when I wrote "Personally, I'd prefer if HMD just gave us bigger lenses and better sensors and didn't spend time on AI, but that's my personal opinion", I was mainly referring to the use of machine learning in the post-shot pre-view stage, as this is the part which stops photos looking natural and starts to make them look overprocessed, AFAIK.

Please don't take any of the above as truth, it's only what I currently think based on my current understanding - if you know different then please explain how it all actually works, I'll be very happy to learn

naufrago

@madbilly

A good read on computational photography:
https://www.digitalcameraworld.com/features/what-is-an-ai-powered-camera

Actually, there are tons of different area of study on deep learning. Here is interesting one;
http://personal.psu.edu/fuf111/publications/intelligent-portrait-composition-assistance-deep-learning-image-retrieval.pdf

madbilly

Oooooh! That will keep me busy for a few nights! Thanks @naufrago

madbilly

I forgot to add this above. In the experience zone there was a demo of how much of an effect OIS has, I took a video; the camera on the left has OIS and the one on the right does not: https://photos.app.goo.gl/m2apzCabjwviQdQ3A

petrus

@madbilly
 I found a nice article. There is a mention of the madbilly
https://nokiamob.net/2019/01/09/hardware-software-multi-camera-phones-and-other-stuff-discussed-by-nokia-community-forum-members-at-the-dubai-event/

naufrago

petrus said:

@madbilly
 I found a nice article. There is a mention of the madbilly
https://nokiamob.net/2019/01/09/hardware-software-multi-camera-phones-and-other-stuff-discussed-by-nokia-community-forum-members-at-the-dubai-event/

I think our @stipe1906 is the "stipe" on NokiaMob? 
 

madbilly

@petrus yes Stipe from NokiaMob contacted me to ask if he could write about my posts and link to them, very polite. It's a very nicely written, well written article, a good summary.

@naufrago maybe indeed! I was also wondering which other prominent Nokia news journalists are on the forum here...

nitheesh

Thanks @madbilly for the useful information. Again you wondering us . I have no words, you explained everything. A big salute for your contribution.

Nice shots. I bookmarked your photo gallery.

Here is my gallery too: https://photos.app.goo.gl/UFmu4exKjzJyxKAA7

madbilly

@nitheesh thanks for the link to your photos!

Which reminds me, we still don't have those photos which were taken of us with Florian. Maybe @edo knows where we can get them? (I'm off topic here, but since I started the topic I'm taking OP prerogative!)

nitheesh

@madbilly yes, that's the only one remaining. Thanks for the reminder. Hope @edo can help us

SeqTre

@madbilly Thanks for sharing.

I was really interested in the discussion you had with that ZEISS guy. You were so much into it that you didn't even notice me trying to tell you that we were leaving back for the hotel.

juho lemmetyinen

Good job! I also have a lot of insights and fun conversations with the HMD staff, but I'm too lazy to update my experiences..

madbilly

@SeqTre I know! Sorry about that

@juho lemmetyinen Oh come on! Please? At least just add in the comments to mine what other things you learnt - we all want to know!

wyk

thanks @madbilly for telling us the secrets from HMD's kitchen... excellent conversation excellent narration.. 
i feel HMD should revamp the camera UI first as it feels cluttered ...when we can swipe shutter button up to launch pro mode so why a different tab for pro needed. In GCam
Google also defines auto exposure..  exposure compensation. . HMD parameters according to user this makes the app stand out from others. 

hughmnyksj

@madbillyThanks for your excellent work here. Folks like you and Steve Litchfield raise the standard of discussion to where it belongs. 
To others who were in Dubai: please do comment and/or add your own stuff.
HMD seems to want to listen to users/fans, although why some of the basic failings of the Nokia 8 have not yet been fixed beats me.
I have no issue with picture quality (I frequently take whole series of photos with the standard camera app and the pixel app); I'm referring to horrible EIS 720/1080 video and the video focus issues. My 8 plays 4k video perfectly, BTW. Just can't edit it in-phone...

jptizon

This is my Review of Nokia 5.1 Plus, that's before Android Pie update..

https://nokiarevolution.com/nokia-5-1-plus-review-is-it-a-top-notch-mid-ranger/

i was a bit disappointed  about the Camera, but after Android Pie update, it tooks better photos, better HDR and a pro-mode.. Low - Light also improve.. kudos to HMD for doing great on updates.. i hope they keep on doing this .

madbilly

Woah! To be compared to Steve Litchfield, that's an honour! Thanks @hughmnyksj

Thanks @wyk glad you enjoyed it!

Thanks for sharing @jptizon that's interesting to hear about the improvement you've seen with the 5.1+ camera after upgrading to Android 9. And thanks for reminding me about your site, I will add it to my RSS reader

nitin kachare

@madbilly hmd can keep it natural images which they does on Nokia Camera i.e.HDR Or Normal Mode, no problem.But They Can add feature like google camera hdr+enhanced where they can enhance images by AI i.e.Machine Learning Same As Google Camera Works.Some Of The Images are very eye catching and beautiful when Google Camera used in HDR+Enhanced.By doing this They can work on both normal hdr and Enhanced hdr.

nokianchetan

@madbilly 👌👌 Very Good 
You are Right HMD needs to focus on better sensor instead of AI , AI on Nokia 8.1 is not doing Well 
See The Difference Below 
In Daylight AI capture More Cold Tones Than the original 
While switching OFF AI capture Original Colours