Nintendo Labo developer interview – Part 2: Prototyping and Development

Interviewees, left to right: Mr Sakaguchi, Nintendo Labo Director / Software Lead, Mr Kawamoto, Nintendo Switch Director / Nintendo Labo Producer, Mr Ogasawara, Nintendo Labo Hardware Lead

Nintendo of Europe Editor: Our chat with the developers behind Nintendo Labo continues in Part 2! 

“Prototype party”

Editor: Now I’d like to dig a bit deeper into the Nintendo Labo development process. I heard that Sakaguchi-san and Kawamoto-san weren’t on the same team when the development process began, but that your teams were later merged. Can you talk a little about how you all came to work on Nintendo Labo?

Mr. Kawamoto: Sakaguchi-san was actually already prototyping before we started working together.

Mr. Sakaguchi: That’s right. At the beginning, I was working on it by myself. At some point four other people joined me and we decided to do something with our ideas.

Editor: And you wanted that “something” to be built around the Joy-Con controllers?

Mr. Sakaguchi: All we had been tasked with at the time was to do something with Nintendo Switch.

Editor: So you wanted to do “something”, but didn’t know what it would be? Sounds like a very uncertain start!

Mr. Sakaguchi: One of the ideas from our first brainstorming sessions was to make scissors using two Joy-Con. We figured they could move like this – you know, like scissors. We mentioned this earlier when we were talking about physical feedback, too, but it was the scissors idea that made us realise that constricting the movement of the Joy-Con, so they could only move in particular ways, improved the data we got from them. We could then write a program assuming that the Joy-Con could only move in these ways, which really simplified the process. The gyro sensors hadn’t been used this way before. There were a lot of ideas from these early sessions, but I was mostly focussed on trying to make new attachments for the hardware.

Editor: What did people think about that?

Mr. Sakaguchi: Most people, including myself, thought it was a unique idea that would never get off the ground.

Mr. Kawamoto: I heard you were prototyping attachments with a 3D printer…

Mr. Sakaguchi: I was! But it turned out that making attachments out of plastic would be too costly, so at a certain point I had to set that idea aside.

Editor: Just to be clear, when you say “attachments”, you’re referring to hardware accessories for the Nintendo Switch system, right? That was your focus at the time?

Mr. Sakaguchi: Correct. Once we joined up with Kawamoto-san’s group we agreed that everyone would bring in their ideas and we’d start prototyping games that worked with them.

Mr. Kawamoto: The whole group was focussed on making something unique with the Joy-Con, and coming up with ideas for intuitive games. We called our efforts “prototype parties”.

Editor: So you started out just wanting to gather as many ideas as you could?

Mr. Kawamoto: Pretty much. We agreed to ignore concerns about the cost and just solicited ideas that were interesting and intuitive. We didn’t provide any more detailed direction than that. Even though I was the one soliciting ideas, I have to admit that it must have been hard for the team to work with such a vague task. (Laughs.)

Mr. Sakaguchi: Yeah, we didn’t want to worry about being realistic at that point, we just wanted ideas. I decided to explore the idea for attachments again. There was just one thing that I kept thinking about… The right Joy-Con has an IR Motion Camera in it, but the camera was a little hard to use in a normal game.

Editor: How so?

Mr. Sakaguchi: It’s a little hard to explain. The camera is plenty powerful, but it’s hard to control. Consider the gyro sensor: if we want to make a game where a character transforms into a different shape when the Joy-Con is tilted, that’s easy enough to implement. But the IR Motion Camera isn’t as simple – and yet we knew that if we didn’t use the camera in addition to the controller differential controls, we wouldn’t be fully utilising the Nintendo Switch hardware, and we wouldn’t be creating something intuitive.

Editor: What was the hardest part about working with the IR Motion Camera?

Mr. Sakaguchi: Well, it’s a camera, so the data you get from it isn’t stable.

Editor: What does that mean?

Mr. Kawamoto: This gets a little technical, but imagine spinning a gyro sensor through the air. In that case the data received is also difficult to parse.

Remember when we talked about how we can’t know how the consumer will hold or move the Joy-Con, which is why constricting the movement of the Joy-Con makes it easier to get reliable data from the sensors? It’s the same issue with the IR Motion Camera. Moving it through the air results in unreliable data. We don’t know how the consumer will move, either. They could step out of the camera’s field of view, or they could stand too far away.

Mr. Sakaguchi: Eventually we realised that we could get the performance we wanted if we enclosed the camera in a box. If the position is fixed then the camera will know where to look for a reflective marker sticker and being enclosed means it won’t erroneously pick up other light sources.

Mr. Kawamoto: We realised pretty early on that we’d have to put it in a box!

Mr. Sakaguchi: Here is the first project we made with that idea.

Editor: What the… Nose picking?!

Mr. Ogasawara: We built a small enclosure around the Joy-Con’s IR Motion Camera. This allowed us to reliably read the motion of objects inside the enclosure. This is the first prototype we made to test the concept.

Editor: I don’t even know what to say. That’s ridiculous. (Laughs.)

Mr. Sakaguchi: It might look ridiculous at first, but we were testing an important idea. You see, we wanted to start with the smallest enclosure we could think of…

Editor: You mean…a nostril?

Mr. Sakaguchi: In this case, the IR Motion Camera inside the box registers movement and translates it to the finger on the screen. This was just a demo, but I remember thinking that if it worked, it would help us pin down whether the box trick would be effective or not.

One funny thing that happened during a test play session: a team member playing with it checked his finger afterward to see if anything had stuck to it. I remember thinking we were onto something there – with the technology working together seamlessly we’d achieved a real sense of immersion.

Mr. Kawamoto: Of course there was nothing to get stuck on his finger! (Laughs.)

Mr. Sakaguchi: When we finished our tests with this prototype, we wanted to try making a really large box next. We had ten people from the prototype team working on it.

Editor: What did you make the big box out of?

Mr. Sakaguchi: For the nose-picking game we used a 3D printer, but even when the printing process goes smoothly it still takes a whole day to produce one prototype.

Since our last project was about discovering how small we could make something, we wanted to see what the maximum size we could work with would be. We decided to go to the packaging material storage room and look for materials. We came back with a bunch of cardboard and used it to make this tank thing.

Editor: So this is where cardboard makes its appearance! What’s the broom-like thing on the top?

Mr. Sakaguchi: It’s a floor sweeper. The foot pedals are hole punchers.

Editor: How were the sensors used in this prototype?

Mr. Sakaguchi: There is a Joy-Con controller inserted in the bottom, and there are reflective balls suspended beneath the pedals that move up and down. The IR Motion Camera just responds to the motion of the balls.

Mr Kawamoto: The tank project showed us how much fun the mechanisms behind the projects themselves can be.

Why we decided to use cardboard

Editor: So you didn’t set out to make anything out of cardboard from the get-go? Ease of assembly was the priority for materials, and that’s what led the team to cardboard – do I have that right?

Mr. Kawamoto: That’s right. A typical 3D printer can produce an object from schematics quickly, but with the pace of our work it just wasn’t fast enough. We were making things, adjusting them, and testing them very rapidly. The 3D printer just couldn’t keep up.

Mr. Sakaguchi: That’s true. But the cardboard tank did break easily, because you had to apply your body weight to the top of it. So at the time I was still thinking that the final product would need to be made out of plastic. The tank also had the right Joy-Con stuck in place, so we weren’t making good use of the gyro sensor, which felt like a missed opportunity. That’s when we got the idea to lift the box off the floor and put it on our backs. This allowed us to utilise the gyro sensors and solve the body-weight issue at the same time. We tried it out, and came up with this robot prototype.

Mr. Ogasawara: We called it a “Carry-Con”.

Mr. Kawamoto: And when they showed me this robot prototype I thought they’d lost their minds (laughs). I mean, just look at the thing.

Editor: (Laughs.)

Mr. Kawamoto:  Not only that, but I don’t think I’d ever seen a game console controller that makes use of a player’s hands and feet. The whole idea was crazy, in a good way (laughs). I was impressed, though, and I wanted to turn this into a real product. There were a lot of barriers to making it a hardware attachment, so I wasn’t sure how to proceed.

Mr. Sakaguchi: We knew we’d come up with something interesting, but we weren’t sure if it would sell. We’d come up to a hurdle, and I remember there were two specific things that got us over it. The first was Kawamoto-san’s idea that we incorporate the “making” process into the final product. The second was, well, we came up with a really boring prototype…

Editor: What was it?

Mr. Sakaguchi: This.

We made this music box, and as things turned out, it wasn’t fun at all. It basically worked the way those spinning barbershop poles work – they have diagonal stripes and the whole object rotates. The sign makes the rotation appear as if there is vertical motion, and we thought it might be neat to use a camera to capture that motion. Then we could calculate the speed and change the tempo of the music accordingly. The way it worked was so cool that we got really excited about it…

But then I thought about it from the player’s perspective, and I couldn’t imagine what they would do with it. It was an attachment for the Joy-Con that was basically a mystery for the player because there was no visibility into how it worked. So for the player it was an experience of a little spinning thing that made noise. I was like, “Who would enjoy playing with something like that?” (Laughs.)

Mr. Kawamoto: In the end, the prototype just wasn’t any fun.

Mr. Sakaguchi: The interesting part about the music box was that we could see how it worked, and the process of thinking through the concept itself was fun – and I realised that those are the very things the customer wouldn’t be able to experience. So we figured out exactly what we had to avoid, and that was huge. If we were going to make this into a product, we had to provide the whole experience to the customer. That’s when we realised that using cardboard and having the customer build the projects themselves was the right direction to go in. It took a potential weakness in the project and turned it into a strength.

Editor: So you decided on the material and the Make part of Nintendo Labo all at once?

Mr. Sakaguchi: That’s right. We realised that if we used cardboard, the user could build the project themselves, they could fix it themselves, they could make alterations to it themselves…and so on. It brought a lot of freedom into the Make process. Even better, the consumer could experience the joy of discovery. These ideas eventually led us to create the Toy-Con Garage mode.

As you can see, the Nintendo Labo logo is designed to look like a box with the lower right corner left open. We designed it like this to show that the box isn’t closed, that you can open these Toy-Con projects up and understand how they work. We wanted it to show that the kits include detailed instructions, but that there are tons of possibilities beyond them.

What can you do with the IR Motion Camera?

Editor: Ogasawara-san, I’d like to ask a bit about the hardware side of things now. Can you tell me how the hardware team came to be involved with Nintendo Labo?

Mr. Ogasawara: We started working together pretty much when the software team came and showed us the “Carry-Con” prototype. What got us to that point was, well, I mentioned in the last interview that I was in charge of the IR Motion Camera.

Editor: You proposed the idea for the IR Motion Camera to Kawamoto-san, right?

Mr. Ogasawara: That’s right. So over in the hardware department we were thinking of ways that we could put the IR Motion Camera to use. We were actually making prototypes of our own, completely unrelated to Kawamoto-san and Sakaguchi-san’s “prototype party”. This was happening separately and completely within the hardware department.

Editor: Another prototype party?!

Mr. Ogasawara: We were calling it “usage research” in my department. Basically we were making a lot of different prototypes, one of which was something we called an “optical attachment.” It looked like this.

In the back we’ve affixed reflective tape to serve as a marker. The Joy-Con emits a beam of invisible infrared light that is reflected back by these markers, enabling the camera to read their movement. By making it like this we can get data when things move without needing any additional electronic components.

Editor: Using the camera to figure out if something is rotating… This is starting to sound familiar!

Mr. Ogasawara: Isn’t it? The first prototype we made with it was this…

Editor: The Fishing Rod!

Mr. Kawamoto: That’s right, but this one operates on completely different principals than the final Toy-Con Fishing Rod.

Mr. Ogasawara: Next, we made this thing. We thought that if we used the optical attachment, we could do all sorts of things with it.

Editor: That looks like cardboard.

Mr. Ogasawara: It is. We made a giant controller out of cardboard and called it the “Big-Con”. The way it worked was pretty simple – you could push the big buttons, and that would move a reflective marker up and down inside the box. Then the IR camera would read the motion of the markers.

This is the first prototype produced with an eye on the building process. I’ve always wanted to show the customer how fun hardware can be. The team got together and thought about how to do that, and we realised that we wanted to actually show the customer how interesting it is to make hardware. We started to think that cardboard was an ideal material for an idea like this.

Editor: You mean that you were all arriving at these ideas independently of one another?

Mr. Ogasawara: Yeah, we didn’t find out until later, but as it turns out we were all thinking about the same things.

Editor: Sakaguchi-san was thinking of “bundles of sensors” while Ogasawara-san was thinking about “optical attachments”? That’s amazing. How did you end up working together?

Mr. Sakaguchi: I was looking for people who were willing to help us with our hardware prototyping, and was introduced to Ogasawara-san. One of the first things we talked about was the nose-picker-con.

Editor: This was during the prototype party, right?

Mr. Ogasawara: Yes. And like I said, my team was already working on prototypes, so we decided to join forces and assist Sakaguchi-san’s team. That was when they showed me their robot prototype, and I was totally blown away. The way they’d used our optical attachment was ingenious, and the software they built around it was really neat.

Editor: It’s like you were destined to work together!

Mr. Ogasawara: It really does feel that way! It was a great coincidence that we were both working independently on similar prototypes, but truthfully the prototypes my team had been working on didn’t have the potential to actually become products on their own. But because I was able to present the IR Motion Camera and the optical attachment that makes use of it to Sakaguchi-san at just the right time, I feel like I was able to make a genuine contribution to Nintendo Labo – I’m very grateful for the way things worked out.

Mr. Kawamoto: I can explain it with a cooking metaphor. Ogasawara-san’s team found new ingredients to work with, and Sakaguchi-san’s team prepared them and plated them up. Together they were able to make a new menu, with dishes no one has ever seen before.

A cross-team collaboration

Editor: OK, just to recap, you started by focussing on making something fun and intuitive, and after a lot of ideas and prototyping you arrived at the idea for a product made of cardboard where consumers could build their own controllers. Is that right?

Mr. Kawamoto: Coincidently, there were a lot of people on the software prototyping team that happened to really enjoy arts and crafts, and that ended up influencing the final product, too. We didn’t choose these people intentionally, either.

Editor: All these people got together and started making things, and that was how you realised that the building process itself was a lot of fun?

Mr. Sakaguchi: That was probably part of it. Our initial prototype brainstorming sessions lasted about three weeks.

Editor: It was only three weeks?

Mr. Sakaguchi: Yeah. We made a lot of prototypes then, ten or twenty of them. Truthfully, a lot of them weren’t any good. It was pretty intense. I guess it was because of that that Ogasawara-san, who made the Joy-Con controller’s IR Motion Camera, ended up becoming “Old Man Cardboard”.

Mr. Ogasawara: That’s what did it.

Mr. Kawamoto: His work has always specialised in electronic technologies, but from now on he’ll be known for cardboard. (Laughs.)

Mr. Sakaguchi: The background of the hardware team members is also pretty interesting. The Toy-Con Motorbike designer was previously responsible for overseeing the Nintendo Switch and Joy-Con hardware, and the Nintendo Switch stand designer ended up designing the Toy-Con Piano.

Mr. Kawamoto: And the Nintendo Switch Pro Controller designer worked on another Toy-Con.

Editor: That is really interesting! I suppose it makes sense that someone accustomed to fine-tuning game controllers so they are optimised for gameplay would bring those same sensibilities to Toy-Con design.

Mr. Kawamoto: We couldn’t have pulled this off without them. The software team alone couldn’t have made all these designs.

Editor: And this wasn’t arranged in advance? The group came together by chance?

Mr. Ogasawara: Yes. We were just lucky, I think, that the team members involved were invested in the project and shared similar ideals. I’m grateful for how it all worked out.

Mr. Sakaguchi: We’ve really only talked about the cardboard designs at this point, but the games themselves are also really well done. The programming, artwork, and sound design all feel great. Play and Discover are just as important as the Make aspect of Nintendo Labo, so I hope that customers will try them out. We even gave the menu button interactions a lot of thought.

Mr. Kawamoto: Every single interactive element of Nintendo Labo feels great. We put a lot of effort and thought into the software that powers it all. The two software art directors came from series like The Legend of Zelda and Animal Crossing. We really were incredibly fortunate to have the team members we had.

Mr. Sakaguchi: A group of programmers came over one day and showed us the Toy-Con RC Car auto-driving mode – they’d come up with the idea and implemented it all on their own. We truly had an amazing group of people working on this project.

To everyone anticipating Nintendo Labo

Editor: Nintendo Labo will be released soon, and customers will finally have the chance to get their hands on it. I’d like to ask just a few more questions, if I may. I’m wondering about the durability of the engineered cardboard in Nintendo Labo. Will replacement parts be available?

Mr. Sakaguchi: Replacement materials will be available from launch.

Mr. Ogasawara: We considered the durability of the materials during the design stage. We were focussed on making it simple to assemble, but of course there are limits to what you can do with cardboard. We designed the Toy-Con projects so that they wouldn’t break easily, and then we performed extensive durability tests where they were subjected to the same motions and actions hundreds and thousands of times. On top of that… Well, this is a bit of a digression.

Cardboard is typically made from recycled paper, at least in Japan. One ramification of the recycling process is that the final product contains all kinds of materials. On the other hand, governments have implemented regulations to ensure consumer safety. North America, Europe and Japan each produce the cardboard sheets for their region to ensure that the material complies with regional requirements.

Editor: On a slightly different topic, I can also imagine children building these Toy-Con creations and getting invested in them. Were they to break in a vigorous play session, the children might want to fix them on their own. How do you feel about that?

Mr. Sakaguchi: I think it’s wonderful. All throughout development we were making things, fixing them, adjusting them, adding new things, and so on. The more we worked on our projects the more ideas came to us, and we’d think of ways we’d want to decorate or customise our projects, and we all ended up with unique, individual Toy-Con creations at the end of it. I like to think that repairs are part of the creation process itself.

The included software even includes tips on how to make repairs. We added videos to the Discover section explaining how best to repair certain things, or mistakes to avoid. There’s even content on decorating tips, and on how to reinforce completed Toy-Con projects.

Editor: I didn’t know that!

Mr. Kawamoto: There are videos on how to replace snapped rubber bands, and how to use masking tape to repair cardboard tears and decorate your projects at the same time.

Editor: Moving on, I imagine there are parents with very young children who wonder if Nintendo Labo is too difficult or complex for their child. Did you learn anything through consumer testing about that?

Mr. Sakaguchi: Yes. The main thing I want to say to parents is that their kids can probably do much better than they might think, so I suggest they let their children try it out by themselves first. Of course, there will still be some children that need more help than others…

Editor: Were there a lot of parents at the tests who couldn’t stop themselves from stepping in and helping their children?

Mr. Kawamoto: I’m sure there are parents that feel like they need to help their children. I also sometimes wonder if… You see, I’m a parent myself. Sometimes I just want to play with their toys. (Laughs.)

Mr. Sakaguchi: I really feel that children have the ability and concentration to do these projects, so I do think it’s best to let them try their hand at building without immediately jumping in and trying to help. There were a lot of parents who were genuinely surprised at how well their children were able to do.

Mr. Kawamoto: We did these consumer tests specifically to address concerns like these. Whenever something was too complicated, we revised the designs. We were always saying, “Back to the drawing board on that part!” We truly did put a lot of effort into polishing the designs.

Mr. Ogasawara: It was a lot of work, but it was always fun.

Editor: How so?

Mr. Ogasawara: It felt so good to successfully address an issue. When people had trouble in one test session, and then didn’t have any trouble after we redesigned that portion of the project, it felt fantastic. It was rewarding, and really motivated me to keep going.

Editor: Did you design the projects and the build time with children’s attention spans in mind?

Mr. Kawamoto: At the Nintendo Labo preview events that were held in Japan, North America, and Europe, we saw a lot of kids who ignored the scheduled break times and went right on making and building – although it’s better to take a break from time to time.

(Editor’s note: The lucky attendees of Nintendo Labo Workshop preview events shared their first impressions here.)

Mr. Ogasawara: There’s no need to try and build it all in one sitting. I think everyone can do these projects at their own pace.

Mr. Kawamoto: When you’re deep in a project and you run into a part that is hard to understand, it’s natural to want to give up at that point. We don’t want anyone to feel that way, so we’ve done all we could to remove any tough steps in the make process.

Editor: Did you notice any differences between people who are naturally good at crafts and people who aren’t?

Mr. Sakaguchi: Not particularly. The difference between individuals was much more pronounced. The Robot Toy-Con takes three or four hours to build, and there was a second-grader who had no trouble staying focussed for the entirety of the build. Once they finished the Make process, most kids said it was a lot of fun.

Editor: Would you say that any kids who have an interest will probably be able to make the projects?

Mr. Kawamoto: Well, everyone is different. Speaking generally, children six and up can probably handle Nintendo Labo projects with a little assistance from their parents. Children ten and up can probably handle the projects on their own.

Mr. Sakaguchi: This is just a personal observation of mine from the tests, but children around eight or nine years old were right in the range where they start to reconsider how much help they want from their parents, and where parents start thinking about letting their children have more independence. Again, everyone is different, but I think that’s a fantastic age for kids to try out Nintendo Labo.

Mr. Ogasawara: Children too young for the making part can still enjoy the games. Even parents will find themselves getting a lot of fresh surprises from the games.

Mr. Kawamoto & Mr. Sakaguchi: This is starting to sound like a product that even children can enjoy, but it really isn’t a product for children exclusively. We made this so it would be fun for everyone.

Editor: Thank you. You must be excited to finally get Nintendo Labo into the hands of the public. Are you looking forward to seeing what kinds of things people make?

Mr. Sakaguchi: I really am.

Mr. Ogasawara: Very much so.

Mr. Kawamoto: I can’t wait!

Mr. Sakaguchi: Honestly though, personally I’m fresh out of ideas. (Laughs.)

Everyone: (Laughs.)

Mr. Sakaguchi: How many ideas can there be left? (Laughs.) I’d like to see someone invent something that is easy to understand, fun to play with, and can only be done with Nintendo Switch – but I’m skeptical.

Editor: We’ll just have to wait and see what the public will come up with!

Editor: This is the end of the second part of our interview with three Nintendo Labo developers. I hope you enjoyed it. Thank you for reading.

Click here for Part 1: The Concept