Futuremark are the industry’s leading provider for PC and mobile benchmarks. Their tools like 3DMARK have been consistently used to gauge performance across smartphones to high-end, multi-GPU gaming desktops. GamingBolt recently chatted with Futuremark’s President Oliver Baltuch on a variety of topics ranging from the latest GPU technologies, DirectX12 to what they are doing in the mobile space and cloud gaming. Check out Oliver’s response below.
Rashid Sayed: To begin, can you please let us know about yourself and the company you work for?
Oliver Baltuch: Sure. I’m Oliver Baltuch. I’m the President of Futuremark. We create 3DMark and PCMark, benchmark tools that help you test and compare the performance of your PC, tablet and smartphone.
Rashid Sayed: Futuremark has been at the forefront of delivering benchmark tools in the last few years. How rapid has performance improved on GPU/CPU side in the last five years?
Oliver Baltuch: Just last year we celebrated the 15th anniversary of 3DMark. We’ve been doing this a long time!
In recent years, I think the most exciting and rapid improvements have been in the mobile space. Last year we released 3DMark for Android and iOS and you can already compare more than 1200 different devices in the app. The difference between the fastest and the slowest handsets is staggering. It really shows how fast mobile hardware is improving.
Rashid Sayed: How difficult is it becoming to benchmark GPU/CPU especially when the performance is consistently improving?
Oliver Baltuch: One of the biggest challenges is the wide performance range covered by a single graphics API. Take DirectX 11 for example. There are cards like the AMD Radeon R9 290X and NVIDIA GeForce GTX 780 Ti that offer phenomenal performance. But there are also laptops with mobile GPUs and CPUs with integrated graphics that support DirectX 11 as well.
It’s hard to create one test that scales meaningfully across such a wide performance range. 3DMark Fire Strike, which is our DirectX 11 benchmark for high-end gaming PCs, is simply too heavy for many laptops and integrated GPUs. That’s why we recently added a new test to 3DMark. Sky Diver is a new DirectX 11 benchmark designed for gaming laptops, mid-range PCs and other hardware that cannot achieve double-digit frame rates in Fire Strike.
"We'll never stop delivering a cutting edge 3DMark experience to core PC gamers, but it was obvious to us that there was a real need for high quality, reliable benchmarks for testing all these new mobile devices and platforms. The result is our latest version, simply called 3DMark."
Rashid Sayed: I wanted to talk about the Dual GPU Titan which comes packed in with 12GB. Do you think that much amount of processing power and compute unit are going to be used in upcoming games?
Oliver Baltuch: The Titan Z is obviously pretty extreme right now, but in the next couple of years, if you are serious about gaming at 4K resolution with a decent frame rate then yes, you are going to need that kind of power. And for convincing and comfortable VR, you really need 120fps at 4K per eye. That’s going to need a serious amount of processing power too.
Rashid Sayed: What can you tell us about 3DMark? What kind of iterations have you made to churn out the most feature filled version yet?
Oliver Baltuch: From the beginning 3DMark has been a showcase for real-time gaming graphics. We love PC gaming and we love playing with the latest, high-end hardware, but new trends are emerging too. Tablets and smartphones are bringing games to new audiences. With Windows, the focus is moving towards tablets, ultra-portable notebooks and new lightweight hybrid designs. All this mobile hardware is improving really quickly and there’s lots of competition.
We’ll never stop delivering a cutting edge 3DMark experience to core PC gamers, but it was obvious to us that there was a real need for high quality, reliable benchmarks for testing all these new mobile devices and platforms.
The result is our latest version, simply called 3DMark. It’s the first to include cross-platform tests for Android, iOS and Windows and the first to include tests for different DirectX levels on Windows. We’re just added the new Sky Diver test I mentioned earlier and we’ll add more tests over time as graphics technology improves, whether in mobile or PC.
Rashid Sayed: Your benchmarking tools have already been used in DirectX 12. What kind of performance gains are we looking at when DX12 goes live next year? Do you think it will have an impact on the Xbox One as well?
Oliver Baltuch: We were very happy when Microsoft chose 3DMark to show some of the benefits of DirectX 12. In their demo, they showed two major improvements over DirectX 11 – a 50% improvement in CPU utilization, and better distribution of work among threads.
With DirectX 12 game developers will be able to create richer scenes with more objects. DirectX 12 also introduces a set of new rendering features that will improve the efficiency of algorithms used in rendering and collision detection.
It’s too early to know the exact performance gains, but when DirectX 12 is ready we’ll be there with a 3DMark benchmark that shows the new possibilities.
"As with all our benchmarks, Servermark will be developed by working closely with our industry partners. As I said, it's early days yet, but I'm sure we'll have more to share as the final specification comes together."
Rashid Sayed: I see that PCMark is compatible with Windows 8. Given that the platform has not really been successful in attracting gamers, what is the demand of PCMark like on Windows 8?
Oliver Baltuch: PCMark 8 is a complete system performance benchmark. It includes a variety of tests based around home, work and creative scenarios. It includes benchmarks using real-world applications such as Microsoft Office and Adobe Creative Suite. There are dedicated storage tests and it can measure battery life, too. And not just for Windows 8. PCMark 8 also works just fine with Windows 7.
The main audience for PCMark is professional users and PCMark 8 is proving to be exceptionally attractive to enterprise, government and industry users.
Rashid Sayed: Can you tell us and describe some of the features about Servermark?
Oliver Baltuch: Servermark is a new benchmark currently in the early stages of development. The aim is to test server performance under a number of common use cases. For example, we are planning tests for mail, file and collaboration servers as well as web and database.
As with all our benchmarks, Servermark will be developed by working closely with our industry partners. As I said, it’s early days yet, but I’m sure we’ll have more to share as the final specification comes together.
Rashid Sayed: Given that you guys are deep into benchmarking, I wanted to know your thoughts about Microsoft’s claim of making the Xbox One more powerful using the cloud. They showcased the tech at Build 2014 and it does look promising.
Oliver Baltuch: It does look promising, doesn’t it? I imagine the challenge will be to use it in such a way that gameplay isn’t compromised if the console is offline, or has a poor connection. It will be interesting to see how developers make use of it.
Rashid Sayed: Do you think the new consoles [PS4 and Xbox One] will catch up to high end PC performance anytime, especially given their ability to allow developers to code ‘close to the metal’?
Oliver Baltuch: The new consoles were already behind high-end PCs at launch, and because the performance of high-end PCs is a moving target, that gap is only going to get wider. What’s more, the ability to code ‘close to the metal’ will soon be available to PC game developers as well through DirectX 12. So no, I don’t see either console being able to match the raw power of a decent gaming PC now, or anytime soon.
"SteamOS has the potential to take Linux and OpenGL into the mainstream for gamers. And if that happens, where gamers are choosing between a Windows gaming PC and a SteamOS machine, then we will want to have a benchmark that helps them compare and choose."
Rashid Sayed: Can you explain to us what Futuremark is doing to improve benchmarking in the mobile space?
Oliver Baltuch: Competition in the mobile space is really intense. I’ve seen figures saying there are now more than 4000 Android devices to choose from. So there are two main ways we are trying to help people choose between all those devices.
The first is by making the best benchmarks bar none. All our benchmarks are created through our Benchmark Development Program. Members include AMD, Imagination Technologies, Intel, NVIDIA, Microsoft, Qualcomm, and many others. BDP members are involved in every stage of the planning and development of each new benchmark. You simply cannot make relevant and accurate benchmarks without that level of industry input.
I’d also say that our approach to benchmarking is more useful for press and end users since our benchmarks are built around the ways people actually use their smartphones and tablets.
For example, 3DMark is gaming performance benchmark. So when we create the tests, we use the same tools and techniques commonly used by game developers to create game-like content that stresses the hardware in the same way games do.
We will soon be bringing PCMark’s application-based benchmarking approach to Android as well. Instead of artificial component tests that are hard to relate to real-world performance, PCMark Android tests the complete device while browsing web pages, watching videos, editing images and writing text.
The second way we’re improving mobile benchmarking is by working to a higher standard of professionalism and accountability. For example, when stories of widespread benchmark boosting broke last year, Futuremark was the only benchmark developer to take action against manufacturers for their misleading practices. We delisted the offending devices by removing them from our performance rankings, which won us a lot of praise from press and end users.
Futuremark is the only developer with publicly posted benchmark rules for manufacturers. And we’re the only developer who backs our benchmarks with public documentation that explains what we’re testing and how we calculate the scores.
Taken together, our deep expertise, wide industry cooperation, and open processes ensure that we produce the best benchmarks for our customers and the market at large.
Rashid Sayed: What can you tell us about Steam Machines? Given that Valve has largely given out confused messaging regarding the same, how are you guys updating your tools for the same?
Oliver Baltuch: Well, we only know as much as you do, but we look at it like this. SteamOS has the potential to take Linux and OpenGL into the mainstream for gamers. And if that happens, where gamers are choosing between a Windows gaming PC and a SteamOS machine, then we will want to have a benchmark that helps them compare and choose.
Rashid Sayed: Have you ever looked into console benchmarking? Or does the industry have no such need of it?
Oliver Baltuch: We talk about it sometimes. Console hardware is not that interesting as it never changes, but trying to test the entire offering, including any cloud computing capabilities that may change over time, might make for an interesting benchmark.
Going back to Steam Machines for a minute. Some of them are almost like consoles. They are small, stylish boxes that sit under your TV and just work. But there are a lot of different Steam Machines to choose from, all with very different hardware inside. I could see us making a console-style benchmark for Steam Machines that would help people choose.
Rashid Sayed: Is there anything else you want to tell us about Futuremark?
Oliver Baltuch: I would like to say that there is a free version of 3DMark for people who have read this far and are now curious to test their own PC, smartphone or tablet. On Windows, 3DMark Basic Edition includes Fire Strike, Cloud Gate, Ice Storm, and our brand new Sky Diver test – everything you need to benchmark your hardware and compare your score with similar systems.
You can get 3DMark Basic Edition from our website or from Steam by clicking on the Download Demo button on the 3DMark Steam store page.