Getnamo – Devblog

There haven’t been any longform blog entries in quite a while, though you may find newer short entries on my twitter: https://twitter.com/getnamo

For the past couple of months I’ve been working on bringing TensorFlow to UE4, enabling the use of state of the art machine learning in Unreal Engine projects. The plugin has reached something I think is ready to try out, but it is very much a work in progress so please check it out and give some feedback if you can!

There’s the plugin

https://github.com/getnamo/tensorflow-ue4

and the examples project

https://github.com/getnamo/tensorflow-ue4-examples

There are pretty detailed instructions at the repositories outlining how to use the plugin with examples.

What examples you ask?

Basic Add/Subtract float arrays

https://github.com/getnamo/tensorflow-ue4-examples/blob/master/Content/Scripts/addExample.py

A simple example that shows how to pass data into python, running tensorflow with the data and then passing results back out to UE4

Mnist Recognition using UE4 textures

https://github.com/getnamo/tensorflow-ue4-examples/blob/master/Content/Scripts/mnistSpawnSamples.py

Maybe you want to recognize hand written digits using UE4 textures? How about drawing shapes in your browser?

Upcoming Pong example

Not quite ready to use yet, but soon there will be a pong AI example, trained using basic q learning

Which one will be the champion?

Try it out and pop any feedback you may have on https://forums.unrealengine.com/community/work-in-progress/1357673-tensorflow

I had the awesome opportunity to join a team with Matt Burri and Mark Bailey for the VRFocus Virtual Reality Challenge. A 24 hour session of frantic coding, meowing, and last minute hacks to get something playable.

Our team cobbled together Catastrophe, a subversive multiplayer game where you have to save falling kittens while waiting to betray your fellow coop partner. Here’s an early singleplayer gameplay video:

The game only really picks up its true form in multiplayer though, which is what we presented at the end of the countdown as the winning entry. VRFocus covered their event extensively, their articles, a recommended reading.

My favorite part of jams is always the end when you get to try what the other teams did; inspired entries, each with their own unique take on the theme, in this case ‘A safe pair of hands’.

Apart from being crazy fun, there were a ton of challenges which we later covered as a presentation during the latest London UE4 meetup.

Check out the slides for more information.

We’re hoping to take this further, keep up to date with the project at our website catastrophevr.com

 

I’m a big fan of web technology. Scripting in javascript makes me jealous that VR development isn’t quite as easy just yet, with this release that development gap hopefully shrinks just a little bit.

For the third London is Unreal meetup I made a presentation on the topic of plugins and chose to implement a useful plugin live during the presentation. A socket.io client was chosen as the plugin, which was finished with a basic Connect, Bind, Emit functions and an On event all interfaced through an Actor Component. Doing this in a small time frame was a daunting task, but something I was certain was doable. While I didn’t meet my target of 15min for the live implementation, within 25minutes the whole meetup was chatting away from their phones with a UMG chat widget in the game engine.

The presentation slides can be found here.

If you’ve haven’t heard of socket.io before, it is a performant real-time bi-directional communication library. There are two parts, the server written in node.js and the client typically javascript for the web. There are alternative client implementations and I’ve used the C++ client library and ported it to UE4. In sum, it makes networking very easy, real-time and flexible.

How does this look like in blueprints?

After adding the new plugin component to your actor of choice

You connect and bind functions of choice

which you can receive as events

and emit equally easily

A C++ version of this setup is available and I encourage you to check out the main forum thread and github repository for more details.

Made a quick experimental port for using the Razer Hydra in VR

This branch of the hydra plugin won’t be maintained, but allows you to try out the VR editor if you don’t have a Touch or Vive lighthouse motion controller

See the experimental branch for details on how to get it running: https://github.com/getnamo/hydra-ue4/tree/vr-editor

Since 4.11 the community plugin I’ve maintained has been included in the engine. Simply enable the plugin in your project to begin using it

Then change your game mode to use the convenience character to see your hands like this

For quick setup: https://developer.leapmotion.com/unreal#quick-setup

Latest documentation: https://github.com/getnamo/leap-ue4

Hopefully you like the updates and convenience blueprint content, post any issues you have to the main forum thread or the master repository.

 

 

Further work on the NexusVR application has required me to find a solution to handle zip files. Having found no suitable alternatives, I decided to port 7zip to UE4. This resulted in the ZipUtility plugin.

Fully multi-threaded and exposed to C++ and Blueprint, it allows for easy extraction and progress update.

From 7zip support for many file formats are available such as Zip, 7-Zip, GZip, BZip2, RAR(decompress only), TAR, ISO, CAB, LZMA, and LZMA86. This and some utility windows file API functions means that it should serve my needs well and I hope you UE4 developers will find good use for it!

Post any questions to the forum thread or the main repository.

During the 2015 Leap 3D Jam, A collaboration with Mac from Borrowed Light Studios resulted in our NexusVR entry

The concept was a DK2 portaling app that allowed you to transit between VR experienced you loaded in your folder. The original goal was a multiplayer social portaling hub, but due to time constraints the multiplayer aspect of it got dropped. What did make into the final version though was a fully enabled chromium browsing experience which you could browse using your hands in VR.

If you wanted to share something you were browsing to the big screen a simple reach into the browsing window (depth interaction) would convert the current page into a data cube. This data cube could be easily transferred to any visible surfaces which would render the link or video.

Mac Cauley’s visuals created a vibrant landing area that will be the core meet up place in the future social version!

If you have a DK2 and a Leap motion, check it out at: nexusvr.io

I’ve recently implemented a fully rigged collision blueprint in my unofficial leap plugin. The way it is constructed allows for you to change both the collision asset and mesh to suit your needs, while retaining the Leap Motion based input.

I’ve also added easy IR passthrough support in the Leap Motion plugin, allowing you to transition from Virtual Reality to Augmented Reality with a simple gesture. Now you can reach for that beer without removing your headset!

The architecture is finally starting to separate the input from the actual output and it should be relatively easy to have other input drive the rigged mesh.

which is the next step, but for now let’s try out the collision accuracy on some Jenga

I may need some practice, but it is great to see that finger collision is accurate enough to do delicate separation. Leap motion tracking has come a long way!

The Myo plugin has also been featured on Thalmic Labs’ Myo Unleashed series. Its a very concise entry on how to add Myo support to your game in UE4 in under a minute. Let me know if it helps!

See the link below for the blog post:

http://developerblog.myo.com/myo-unleashed-gesture-control-in-the-unreal-engine/

How would you like to climb in VR with kinetic energy from your swinging added to your jump?