The tradeoff of rendering or filming motion blur at finite refresh rates is the audience can move their eyes to follow an object moving around the screen. In real life, that causes the object to become sharp. So, you either need to track eye motion and blur according to relative motion, or do no motion blur at an infinite refresh rate. Neither is practical with todays's technology, so it's always going to look wrong. A good director or game designer will choose shutter rate or render blur according to how they expect the audience eyes to be moving.
However, the audience is used to shutter angle as part of the visual vernacular (e.g., narrow shutter angle for hyper edgy rap videos, long shutter angle for dreamy retro vibes). If rendered content can't speak the same visual language, a tool is missing, regardless of framerate (up to a point... At 400Hz, I'd be impressed by someone really seeing the difference).
What's interesting about rendered content is that it can extend that (e.g., a shutter angle beyond the duration of a frame), playing with something we thought we had a handle on.
I first learned about shutter angle when reading about its use in Saving Private Ryan; the narrow 45 and 90 degree shutters made the opening scenes much more visceral with so much flying through the air.
> At 400Hz, I'd be impressed by someone really seeing the difference.
Trivial. Drag your white cursor quickly across the screen against a black background. You will see clear gaps between the individual cursor afterimages on your retina.
Double the FPS, halve the size of the gaps. On a 240Hz monitor I can so clearly see gaps such that if they were halved the gaps would still easily be visible. Ergo 400Hz would still easily be distinguishable from continuous motion.
To put numbers on this, consider a vertical line 1 pixel wide moving across a 4K screen in 1 second (that's not even that fast). At 480Hz that's a shift of 8 pixels per frame. So you'd need at least 8x the framerate for motion of this line to be continuous.
Another example is multiplexed LED 7-segment displays. These distort in different ways depending on how you move your eyes. Even 400Hz is too low to display a typical one realistically. If you use motion blur you'll lose the eye-movement-dependent distortion of the real thing.
I read a study that put people in a dark room with a strobing LED and told them to dart their eyes left and right. 1000Hz was the limit before everyone stopped seeing glowing dashes and saw a solid line streak instead.
I was researching this because I was wondering how fast you can make LED flicker for lighting effects before it looks like constant brightness.
I found most of the information on Wikipedia[0], and the limit seems to be at about 80hz, but together with movement, some people can see stroboscopic effects up to 10khz.
I’m reminded by an old Microsoft input research video, where 1ms latency response is what is needed for the most lifelike response for touch drawing on a screen: https://m.youtube.com/watch?v=vOvQCPLkPt4
That's definitely not the limit. At 700Hz, a 700px wide screen with the 1px line crossing it every 1s would qualify as reaching the limit in that situation. But, speed that line up so it crosses every 0.5s, and it's no longer good enough. You've introduced an artifact - the object looks like multiple copies now, equally spaced apart by 1px gaps. It's not a smooth blur. The display never displayed a gap, but human eyes merge together an afterimage, so we see multiple frames at once over the last ~1/30s. Now go to 1400Hz, but double the speed of the line so it crosses in 1/4s. Now you need 2800Hz to eliminate the artifact. Or, you can artificially render motion blur, but then it won't look right if you eye follows the line as it crosses. So it's also a function of how fast your eye muscles can move across the screen. Thirdly, we can't limit ourselves to a 700px screen - a screen filling the human field of view would need to be 20-30 million pixels wide before one can no longer resolve a 1px gap between two vertical lines. There is eventually a completely artifactless limit, but it's way higher than 700Hz. Of course, 700Hz is nice, and if you fudge the criteria (how often do you see a 1px line moving across your field of view at high-speed in real life) you can argue it's good enough.
> Trivial. Drag your white cursor quickly across the screen against a black background. You will see clear gaps between the individual cursor afterimages on your retina.
That's the outcome of aliasing, not of the FPS limitation itself. You could analytically supersample the motion in the time domain and then blur it just enough to remove the aliasing, and the distinct images would then disappear. Motion blur approximates the same result.
Yeah but if your eyes were tracking it smoothly, it would not appear blurry. You could try to approximate _that_ with eye tracking but achieving such low latency might be even harder than cranking up the FPS
If your eyes were tracking the motion smoothly, the moving object would not appear blurry, but the static background absolutely would. So you'd need to apply anti-aliasing to the background while keeping the object images sharp. (Similar to how a line segment that's aligned with the pixel grid will still look sharp after applying line anti-aliasing. Motion tracking here amounts to a skewing of the pixel grid.)
"Pixels" and "frames" are the exact same thing analytically, only in the spatial vs. time domain. This is very much an instance of aliasing, which is why blur happens to correct it.
Alright, but supersampling would exactly achieve another approximation to (real) motion blur, but still suffer from the same issues (like not becoming sharp when tracking with your eyes).
Can you recommend any resources on how shutter angle is used to communicate different messages in film? The Wikipedia article only gives some very basic examples (short shutter angle when you want to capture particles in the air or for action scenes).
An alternative to infinite refresh rate is to show each frame only a fraction of the normal frame time, i.e. to quickly "flash" the individual frames and showing a black screen otherwise. This reduces the maximum screen brightness, an it requires a minimum frame rate to avoid flicker, but it reduces the type of "eye tracking blur" (persistence blur) which isn't present in real life. To be precise, to completely remove the tracking blur you would need to flash each frame only for an infinitesimal time period, which of course isn't realistic. But VR headsets do use this flashing/strobing approach.
By the way, this is also the reason why CRT and Plasma screens had much better motion clarity than LCD or OLED. The former flash each frame for a short time, while the latter "sample and hold" the frame for the entire frame time (e.g. 1/60th of a second for 60 Hz). 60 FPS on a CRT looks probably more fluid than 120 FPS on an OLED.
Another option for games is to indeed add a lot of frames by using reprojection techniques. This can approximate the real camera movements without needing to render a ton of expensive frames in the engine. This also is already used in VR, just currently not at overly high frame rates. This great article goes into more detail:
Something like 1000 FPS with reprojection are apparently quite realistic, which should solve the problem of tracking blur without reducing screen brightness.
> Something like 1000 FPS with reprojection are apparently quite realistic, which should solve the problem of tracking blur without reducing screen brightness.
Is a 1000 FPS screen more realistic than a screen capable of the higher maximum brightness needed to compensate for black frame insertion though? HDR screens are already a thing and you could already gain persistence improvements there for LDR scenes without needing any new hardware by always driving pixels at max brighness but only for a reduced time depending on the target luminance.
Or just reduce the ambient light enough - I even run even my LDR monitor at 10% brightness.
As far as I understand, the only thing that matters for reducing unrealistic "tracking blur" is indeed how short a frame is displayed on screen. Which could be achieved by strobing / black frame insertion or by increasing frame rate. Or even both, as in some VR applications. So the effect should be the same, except for the brightness thing, which may not be a problem if there is a lot of screen brightness headroom anyway. For HDR LCD TVs the LED backlights get quite bright actually. OLED not so much.
One advantage of higher frame rates (as opposed to strobing) would be quicker input responses to, e.g., moving the camera. That's not overly important on a normal screen but quite significant on a VR headset where we expect head movements to be represented very quickly.
Low persistence only gives perfect motion when your eye motion perfectly matches the object motion[0]. For all other motion, low persistence causes false images to be generated on the retina, which can be seen as "phantom array effect"[1].
I think interpolation up to several kilohertz is the best solution, preferable starting from a moderately high frame rate (e.g. 200fps) to minimize the latency penalty and artifacts.
I think the second effect is quite small compared to the first because CRTs are generally good at the first but don't suffer much from the second, as far as I'm aware.
CRTs suffer strongly from phantom array effect, but individual sensitivity to this effect varies. The people who notice it are the same ones who complain about PWM lighting.
Moving objects can become even sharper if each frame is displayed in a fixed location for a shorter period of time (reduced MPRT), preventing eye-tracking motion from smearing each displayed frame. This can be achieved through CRT/OLED scanout (often rolling) or LCD backlight strobing (usually full-frame by necessity). Unfortunately displaying each frame for a short time is unbearably flickery at 24 Hz (so movie projectors would show film frames 2-3 times), just barely tolerable at 50 (to the point some European CRT televisions frame-doubled 50 Hz video to 100 Hz, causing objects to appear doubled in motion), and ideally needs 70-75 Hz or above for optimally fluid motion and minimum eyestrain (which can't show 60 FPS recorded video without judder and/or tearing).
> to the point some European CRT televisions frame-doubled 50 Hz video to 100 Hz, causing objects to appear doubled in motion
Hmm, this is not accurate (or I don't understand what you mean). 100Hz CRT TVs available in the 90s/00s did not interpolate frames to get smoother motion – they only existed to reduce flicker. I think such TVs also existed in NTSC markets (120Hz)?
Anyway, ever since the late 00s, pretty much all the TVs you can buy from a store do come with an interpolation algorithm to artificially display a higher frame rate image (e.g. 100Hz/120Hz) from a lower frame rate source (e.g. 23.976/24/29.97/30/50/59.94/60 fps) – which (personal opinion) looks terrible (and can be turned off from the settings – but the default is always on). This is an interesting side tangent when it comes to motion blur, because the blur is prebaked in the input signal and cannot be easily removed. Thus, the end result always has an artificial look.
For instance, if the source material is shot in 24fps with the typical 180 degree shutter angle, each frame spans 41.6ms of which the shutter was open for 20.8ms. Then your TV interpolates that to be 96Hz or whatever. However, the individual output frames still look (or, with the added artifacts etc., mostly look) like the shutter was open for 20.8ms per frame. However, each frame now spans 10.2ms which is a shorter time than the shutter speed!
As far as I understand, you get doubling not because of any interpolation but because you track objects with your eye. That means the tracked object is in rest in your vision. Which means the screen actually moves relative to your vision. Which means the object smears, where the pixel length of the smear is equal to the pixel distance the tracked object moves in the time the frame is visible on screen. On CRT there is less smear, because the frames are only flashed for a short time on the otherwise dark screen. But if the FPS are not equal to the display Hz, e.g. a half, the tracked object appears on multiple, e.g. two, locations on the screen. Because you show the frame twice, and if you track an object, the screen has moved (relative to your vision) for a few pixels by the time you show the second frame. This happens e.g. if you play a 30 FPS game on a 60 Hz CRT, or when you watch 50 FPS content on a 100 Hz screen. Though the effect isn't usually as dramatic as it may sound here. Doubling is also not as bad as smearing, e.g. in the case of scrolling text.
Double images are already present in NTSC/PAL due to interlacing. So when doubling the Hz of PAL on a CRT, you get a quadruple image. Though this probably still looks better than non-interlaced content on LCD/OLED, which exhibits smearing instead.
Double images are only present in converted film content, which in the case of PAL is done by speeding it up from 24fps to 25fps, and splitting each frame into two fields. In the case of native PAL video content, each interlaced field shows image data from a different time, so you don't see double images.
In case of PAL video we don't see two duplicate frames, but we still see two different frames (fields) at the same time. That's the "double image" I meant.
On an interlaced display, the two different fields are not displayed at the same time. If you merge the two fields into a single non-interlaced frame and display it on a progressive scan display you might see a double image, but that's not inherent to interlaced video.
Exactly, while doing background research for this I read a really neat paper that describes what you're saying, and offers a solution based on trying to predict the viewer's eye motion: Temporal Video Filtering and Exposure Control for Perceptual Motion Blur (Stengel et al., 2015)
You can't properly make a moving image sharp at a finite refresh rate (though you can approach it for sufficiently high ones), because the object can only move in a temporally juddered path, which doesn't match the eye's movement.
