I wish to create an interactive experience with Unreal Engine. The environment will be set in the present day, but with a number of futuristic gadgets to use.

Milestone 1: Create an interactive environment using existing environment modules

- Use environment packs to set up an environment

- Remove existing lights, shaders, effects and materials

- Apply my own lights, shaders, effects and materials

- Export a Windows executable, and Mac (optional)

My first step is to see how to export to a windows executable because I am currently using the MacOS platform.

In later assessments, I will explore the use of Quixel Megascans and may create my own assets.

The demo map of a small town that I downloaded is shown below (I did not create this, I just downloaded it). 

The next step would be to import a mesh from the 'town' asset library into my basic scene, and explore what the asset is capable of in terms of customisation.

I was able to export the textures from the 'uasset' file format to a targa (TGA) format. The list of texture files for the PolygonTown pack is shown below:

There are a total of 12 textures in the PolygonTown pack. This is also the case for the PolygonCity pack (12 textures). I exported them to TGA so that I can modify the texture files.

As a small modification I made, I changed the US flag to a Hong Kong flag. This required making changes to all 12 textures in the PolygonTown pack, at the exact same location on the file. I did this by recording a Photoshop Action and playing it back for each file, for consistent results. The Actions are shown below:

The minor modification I made (as a test before going on to make bigger changes) is shown below. (The modified image is shown on the left, while the original image is shown on the right- the bottom right corner shows the difference whereby the flag is changed)

The result in the scene:

The next steps will be to make further modifications to the textures.

I would like to incorporate this into my project somehow. Perhaps I will create two or more sections on the map that can only be accessible via this portal system. I will need to further refine the mechanics behind this door to support accessing multiple locations. Since I have the PolygonTown pack and PolygonCity pack I think I will be able to create a residential area and a city area.

I also added more functionality to the door. There are now currently three locations that the player can travel to using the portal (home, town, and city). As the player goes near the door, instructions appear on the pillar next to the door asking the player to press one of three keys (1, 2, or 3). In the picture below, the player is already at home, so if the player presses the '1' key to select Home, a red cross symbol (x) will be displayed. Pressing '2' or '3' (valid locations for the portal) will display a green smiley face (:-)) on top of the pillar.

The next step will be to create a town and a city. I anticipate they will be placed on the same map, which would take up more memory, but allows the portal to work with the different places.

I created this simple low-poly town that contains most of the building types in the PolygonTown pack. Later on I might add some additional details there. I am now working on creating a city using the PolygonCity pack.

The next step is, as Ivan suggests, to look at lighting and effects. The challenge here would be to create emissive materials for the existing assets, since I am not able to access the original FBX file on the meshes, but exploring this option will soon reveal whether this idea is possible or not. Having said that, the city assets have the option to switch on 'night mode', which involves using emissive materials already set up as an option. Effects are more likely possible, but will involve a greater learning curve. In either case, I will try to explore both. I will add more detail to the city later, but for now it is large enough to be explored as an interactive experience. 

I created a night-time scene for the town and the city. This was achieved by turning down the directional light to 1.0 lux, changing the sun position to -1.0, and turning up the brightness of the stars. The lighting on the buildings and other assets were achieved simply by selecting the alternative 'emissive' material on each of the assets.

I also explored effects using the Niagara system. I created some basic rain effects (including rain and rain splash) to achieve the following look. I had to limit the area that the rain covers to a small section due to performance considerations.

Finally, I added some green odour (in the form of particles) to the back of the garbage truck, as a simple implementation of effects.

The next step will be to refine lighting and add further effects to the environment.

The green ones represent New Territories taxis while the red ones are for the main city (Kowloon and Hong Kong island) taxis. Therefore, as representation within the world I am creating, the 'Town' taxis will be green, and the 'City' taxis will be red. The original, US design is shown below:

In order to create the Hong Kong taxis I needed to change the texture in the number plate, the side logo, and the top 'taxi' text. I used Photoshop to achieve this effect. The end result is shown below:

The limitations are that the top half of the chassis (white in real life) and the bumper (black in real life) cannot be changed. This is because the UVs are pre-set such that the entire chassis has the same colour. When I create my own assets I will be able to customise the look a lot more than what I can do currently with the asset packs.

I have also included below a video of the rain effect I created (previously I only showed a still image).

Limitations include having to create 'exceptions' for cel-shading where the materials have emissive properties (such as street lamps and lighted-up buildings); and the seamless portal implemented earlier does not show the cel-shaded or outline effect when looking through the portal.

The concepts to create the cel-shading is somewhat technical, and the black outlines even more so (it uses Laplacian Edge Detection), so it was good to have the tutorials to explain the process of creating the effects.

The thickness of the outlines can be tweaked, and the cel-shading colour gradient (cel-shading removes the realistic gradient of colours and replaces it with stepped colours) can also be changed, later down the track if needs be.

This filter is quite unique, but required some HLSL (high level shading language) coding. Still, since I was just following a tutorial, it was achievable with relative ease. I implemented this filter as a way to understand how I might implement other filters on my own in the future. One difficulty was when Unreal engine got updated to the newest version, it no longer supported the creation of 'Shaders' folder in the project file structure by default. I had to follow advice from a forum to reapply this folder accessibility within my project.

I also tried to reimplement the seamless portal (but without the teleportation) without revisiting the tutorial, as a learning challenge. I was finally able to get it to work, placing a 'portal' between a drawer and a time tunnel. Furthermore I added a red ribbon effect, using Niagara, in the vicinity of the drawer to indicate that there is something worth visiting there.

As can be seen in the image below, there is now a portal to the time tunnel from the drawer (time machine included), though I have not opened access to the time machine just yet.

I also started playing with creating my own assets, which will form a part of milestone 2. This was done in Maya and imported into Unreal engine. (TM-01 stands for time machine number 1). It was a challenge getting the scaling right at the outset within Maya, so the scaling was performed after it was imported into Unreal.

Something I learnt was that I was wrong about the cel shader not working for emissive materials (and having to create an exception) because the post-processing material used emissive colours which replaced original emissive materials. That was not the problem at all, as I discovered when implementing the Kuwahara filter and successfully retaining the emissive properties of the street lights even without creating an exception. I will look into this a bit more and see if I can figure out what the problem might be with the cel-shader.

I am glad to have tested compiling of the program to an executable, because even though I tested it at the beginning of the milestone a lot has changed and it took a lot of hoop-jumping to finally to get it to compile. There were many obstacles for both compiling for Mac and Windows. For example, for Mac I made a small error in the C++ code, though the debugging messages were so cryptic it was hard to locate the error. I also pressed the 'Clean' button on Xcode, which turned out to be a bad idea because it removed not only the binaries for my project, but also for the Unreal Engine as well. I had to reinstall Unreal Engine, which would have been fine except it then said I did not have enough space on my hard drive even when there was a lot of free space there. As it turned out, after much 'googling', it was because of my Apple Time Machine backup giving local snapshots which had to be deleted. For windows, I had difficulty opening up the project, which involved me installing Visual Studio 2019  trying to look for error messages. As it turned out all I had to do was to remove the C++ source files and place them back in. In sum, the solutions were so far from what I could have imagined to be that it took a lot time to figure out.

I also added a few more new features to the interactive experience. I changed the projectile from a yellow ball that bounces around to a cloud that explodes upon impact with another object in the scene.

I also added Mixamo animations to the characters so now there are people walking around the town. To achieve this I had to create a spline that goes around the entire walking area, use Mixamo by uploading an FBX file that fits in with the characters in the asset pack (it was provided by Synty Studios, the maker of the asset packs), downloading the Mixamo-generated FBX character animations, and do some retargeting from the imported animation to the skeleton of the townspeople. From there I added a 'reverse' functionality so that they can either be walking clockwise, or anticlockwise.

Even though the time machine is still incomplete, I want to share some progress because I do not know whether I have time to complete the entire feature before end of Milestone 1. What I implemented is the ability to go into the time machine, get it to start moving, and exit back home. It currently does not travel to the map representing a different time. To achieve this, I rotated the time machine object around a torus-shaped tunnel, so it can go on forever if it wants to. The pivot (to use 3D animation language) for the time machine is in the centre of the torus. Below is a video demonstrating the implementation:

Throughout the process of rigging these characters, I have found that the new Unreal Control Rig (which only came out last month!) uses quaternions for rotations. There was one element which required some additional manipulation (the cogs that drive the ear rotations) and I found that to be quite challenging. It's just a small component (hardly noticeable), and it's kind of working, so I might leave it for now even though it's an artificial workaround to the problems I've been having with it. Technically if the rig is perfect then there shouldn't be any artificial tweaking required, so I guess I still have a lot to learn in terms of rigging. The next step will be to test the rigs to see if they function properly.

I have also found that the textures I used for the robot on the left image takes up a lot of computing power because it's high resolution, so I will be removing the nice textures when animating to avoid slowdowns of the computer.

Synopsis:

Captain W and his sidekick CR are headed to Mars for a day trip. W sees that the trip costs 2 coins per person, but he only has exactly 2 coins, while they are two people. An idea strikes W, that if he shrinks CR to a small size it can become a part of the luggage. He does exactly this and heads off to Mars while only paying for one person!

The next step will be to create some assets that are required. For the environment I will be using an existing one from an asset pack, called SciFi Hallway from Epic Games.

Since I already started animating Captain W, I thought I would finish it using the manual method. Also the robot requires manual animating because it has quite a unique rig. So it looks like I probably will not use mocap to animate my characters. To clarify, I did place the joints and skinned the robot using Maya (Captain W was pre-skinned), I simply had to set up the control rigs for the two characters (which could not be imported from Maya because they use separate systems). For virtual production, I tried using my iPad to control the cameras but it would not work properly, but I will give it a try again some time later. I'm also not sure if my computer can handle playing the animation back in realtime as I am unable to use the GPU to run Unreal Editor on Mac (might be specifically a Mac issue as I saw on some forums that other people had similar issues). Here is the first draft of Captain W's animation:

My experience so far with Unreal to animate is that it has some similarities with Maya, for example it has a graph editor, but in some cases it is really quite behind Maya in terms of as an animation package. The selection of the controls sometimes fails and you have to manually change the numbers (translate, rotate) to move. There's no local rotation in the viewport as far as I can tell so once again manual changes to roll, pitch and yaw are required. I guess it just takes longer to achieve the same effect.

I added some more elements to the scene, such as the robot animation and fixed some parts of the captain's animation. I also changed the materials on the counter, and added the 8k textures (Quixel Megascans) to the robot. I have found that once you start animating it's not possible to change the rig in the Level Sequencer. The rig is updated but the part of the program where you add keyframes is not updated so I am unable to move the eyes of the robot even though I have the rig set up for the eye movements.

There's still a lot more tweaking I can do, such as changing the camera movement, making the animation more precise when the captain is holding the robot and putting him in the box. More robot animations will be required as it is currently quite static. I will also need to test out the virtual camera again and add sound (dialogue, music and sfx) to the scene to make the clip more complete.

The SFX track is as follows:

The dialogue track is as follows:

When you combine all of those together, it makes more sense and becomes more interesting. Here is the combined sound file:

I used 'Freesound' and 'Youtube Music Library' for the SFX and music. I recorded my own dialogue.

Anyway, this is the animation with exposure adjusted: