Ask the Developer Vol. 5, Nintendo Switch Sports–Part 2
Content pre-recorded in accordance with current COVID-19 health and safety guidelines.
This article has been translated from the original Japanese content.
This interview was conducted before the game was released.
Contents
2: Beyond 90% detection
Part 2: Beyond 90% detection
You mentioned earlier that it was not easy to implement on the Nintendo Switch system what was achieved on the Wii system. So, I would like to ask the programmer Okane-san, what exactly are the differences between the Wii system and the Nintendo Switch system?
Okane: First of all, Wii Sports used Wii Remote™ (4) as a controller, and we incorporated it into the game while thinking about what gameplay could be done with the Wii Remote. But with the Nintendo Switch system, it was the opposite. Initially, the idea of playing sports by swinging was already decided, and we had to figure out how to implement this idea with the Joy-Con controller. As a programmer, this required a lot of trial and error.
(4) The primary controller designed for Wii system home game console, released in 2006. This controller has a vertical shape, similar to that of a TV remote controller, and was designed to enable intuitive and easy-to-understand operation at a time when Nintendo was aiming to expand the gaming population.
To the untrained eye, you would think that since you swing both the Wii Remote and Joy-Con controllers, and they should contain a sensor for that purpose, it could be achieved easily....
Okane: If you hold a Wii Remote, it has a certain amount of weight and size. When players play Tennis or Bowling with this weight, it enables them to control the Wii Remote controller naturally with just the right amount of swing for the game. Therefore, it was easy to guess how the players would hold and swing it, and as a programmer, it was relatively easy to detect how it was swung as well. But the Joy-Con controller for Nintendo Switch is much smaller than the Wii Remote controller. So it varies a lot from person to person in the way they hold and move it.
Certainly, when the controllers are smaller it is easier to swing the arms freely, and I understand that they can swing in many different directions. But how does this actually affect the game developing process?
Okane: It is difficult to put into practice the motion detection: "if you swing like this, it should react like this," in a way that satisfies players.
In a way that satisfies players? Does that mean making sure that things like, "I thought I swung it, but it didn’t get recognized," won't happen?
Okane: Yes. Also things like, "I thought I swung left, but it went to the right." As we progressed, we could implement the detection so that about 80 to 90% of the time, it reacted as expected, at an early stage. However, applying the final polish to motion controls so that they feel natural and completely satisfying when you use them is very difficult. Both Shimamura-san and Yamashita-san mentioned earlier "aiming for the world’s most easy-to-invite to, motion-based game.” To achieve this, it's critical to work on detection beyond 90%.
What were your thoughts on this, Yamashita-san? Did you feel that you could achieve the adjustments “beyond 90%"?
Yamashita: Well, it was tougher than I thought. This is because when we actually made the game and played it with a group of people who had played the previous series, they were able to play it without too many problems. However, when we invited someone who had never played this game before to try it, the ball did not fly in the direction they expected at all. I can understand "the grip is different," or "the swing is different," but there were times when people said things like, "this is not how you move in the actual sport!"...
Not how you move in the actual sport? What does that mean?
Yamashita: Take volleyball, for example. In real volleyball, you don't move your arms and wrists when bumping. Once in the bumping position, the player keeps the hands in a fixed position and returns the ball, using the knees to push through. This is the official way of bumping in volleyball.
I see.
Yamashita: But if you've never played volleyball before, you'd think they were launching the ball in the air by swinging their hands and arms up, wouldn't you? However, the programmer in charge of developing Volleyball was an experienced volleyball player. So, the programmer had the "official bump" move required in the game as well. When we tried the prototype with everyone, as expected, everyone swung up their hands holding the Joy-Con to bump the ball. Strangely enough, even those with volleyball experience did the same move. But the program was designed on the premise to "keep your hands still," so the ball would fly in the wrong direction. In other words, the programmer, who was an experienced volleyball player, created a game mechanic where the player had to keep their hands stationary, set the angle, and bump...
It sounds like it's hard to control unless you know the gameplay.
Yamashita: That's right. It's fun when you know how to play with that gameplay. “Oh, so authentic! This is just like volleyball!” But when we tried it together without any explanation, almost everyone swung their arms up. We all thought, “What are we going to do with this!?” So, we get to a point where we think this is going to have to be a game mechanic where the player needs to swing up. But then the programmer in charge, who has real volleyball experience, will be concerned that the volleyball-ness may be lost.
But even when people with actual volleyball experience played, they were still swinging their hands up, weren't they?
Yamashita: Yes, they were. Considering that, the only way to make a game that people who know nothing about it can play naturally with a quick glance, is to make it a gameplay where players swing their hands to bump after all. ...And so, after some discussions, all the methods to identify the movement were reworked.
Regardless of whether it is the actual move or not, it is indeed easier to understand if the ball flies in the direction that you expect it to.
Okane: If the game is to be played for a long time and thoroughly, it may be an option to do it with authentic and correct volleyball form. However, I felt that it would be difficult to convey the fun of the game in a short period of time, while at the same time having the player read the operation instructions carefully upon playing.
Are such adjustments only in Volleyball? Have there been any adjustments in other sports as well?
Yamashita: There were fewer sports that actually went smoothly. (Laughs) In the planning stage, there were already some kinds of adjustments necessary for all the sports. For example, Badminton. The appearance is similar to Tennis from the previous games in the series. At that time, in Tennis, the direction you hit the ball back was determined by the timing of the swing. If you swung at an earlier timing, the ball would fly to the left, and if you swung at a later timing, you could hit the ball to the right.
If you try to put that same concept into Badminton, and ask someone who doesn't know the premise to play, they may say, "I don't know why the timing of the swing changes the direction in which the shuttlecock flies!" Shimamura-san and I know that it is not the direction of the swing but the timing of the swing that determines the direction in which the shuttlecock (feather) flies, since we know the method of the previous titles. But those who played without knowing this premise, will say that it just doesn't feel right.
Was it not possible to actually recognize the direction in which the Joy-Con controller is swung in Tennis and determine the direction in which the ball is hit according to that recognition?
Yamashita: When you actually play it, that method is very difficult. When we try to figure out the direction of the swing and use it to determine the direction in which the ball flies, it makes the controls unstable, and the ball doesn't fly in the direction you expect it to. Even when Wii Sports and Wii Sports Resort were being developed, I was told by the programmers, "Please don't make it like that.” However, there were suggestions from the staff members who were not familiar with the trial-and-error process at that time saying, "We want to recognize the direction in which the shuttlecock flies by the direction of swing."
Okane: I do understand that badminton is a sport in which the shuttlecock will fly in the direction that you swing. I knew it was going to be difficult since it had already been experimented with over and over again previously in the series.
Yamashita: As a director who has experience working on previous titles, asking them to achieve it when you know it's technically difficult was tantamount to "losing." So, the director has to solve the problem with ideas on how to make the motion controls feel natural, and not through technology. I was saying, “That's what directors do!” But eventually, I had them solve it with technology. However, there was no way it would have been so easy.
In this title, how did you overcome what seemed technically impossible at the time?
Okane: We just did a lot of trial and error. In terms of new technology, we tried deep learning (5), which was not available when Wii Sports was developed.
Deep learning?
Okane: For example, we asked various people to "swing as if you were hitting to the left," and "now swing as if you were hitting to the right," and repeated the process over and over again to acquire a large amount of data. It is a technology that detects if the player is trying to hit the ball to the right (or left) by statistically analyzing that acquired data. So by incorporating such technology, and experimenting with this and that, we tried to bring it up to a level that everyone could agree on.
(5) Machine learning technology captures large amounts of data, such as images and video, into computers and analyzes the characteristics of the data.
Yamashita-san mentioned earlier that it is very difficult, but Okane-san, how did you feel about the direction of the swing determining the direction of the ball?
Okane: I thought it would be difficult. Deep learning is an excellent technology, but it is not magic. If we can get enough data "from the beginning of the swing to the end of the swing," we may be able to estimate the correct direction. But if the ball reacts and flies out after the player has finished swinging, the players would feel it's not responsive, and it's one step behind. So then, if we try to use data from just before the end of the swing, the correct movement cannot be estimated because there won't be enough data this time. In other words, there is a trade-off between aiming for "accurate detection" and "satisfactory response as a motion control." So, we made adjustments over and over again tenaciously saying things like, “I think it's somewhere around here!... No, maybe it’s somewhere around here!”
But with the accumulation of these steady efforts, the 80-90% you mentioned earlier got closer to 100%, right?
Okane: Yes, so I would say that we could reach about 90 to 95% detection rather smoothly. But the last few percent was really tough... For example, when a player serves in Badminton, and even if the shuttlecock flies as expected nine out of 10 times, if the player misses even once, they will think, "Hmm, it's hard to control..." So, even though it's impossible to reach 100%, it is very important to work on this last small percentage, even if only by a little. This is no longer something that can be solved with the latest technology, so we all worked hard by getting as many people as possible involved and making detailed adjustments.
Shimamura: Every day, we would hit 200 times each to the right, left, and center to see how the latest adjustments worked. My right arm was so pumped and crab-like. (Laughs)
I guess developing a motion-based game is hard when it comes to testing, too...
Yamashita: It is a sequel to the previous games in the series and also a motion-based game, but as explained, it couldn't be just a simple port. I think all the sports were close to being rebuilt from the ground up in a way that was adapted to the Nintendo Switch system.