Messing with photogrammetry

I was trying to apply photogrammetry to a game asset out of curiosity, I have used photogrammetry in my work experiences to build high fidelity maps and was curious about further building the skill for game asset creation. I stumbled on this tutorial:

https://www.agisoft.com/index.php?id=38


At work, I had it easy because I had the professional version which permits adding markers to the pictures to help with alignment. At home, I have the home license which doesn't have this feature, so while trying to apply photogrammetry at home I needed to learn how to use the basic tools more effectively. It looks I've been taking the background for granted and thinking that the solver would address this for me. With some experimentation, it wasn't working. A few lessons learned:

• I tried resizing the region (this was key) of interesting after the photo-alignment on 400 photos, I got a useful result! 
• I tried a few more runs with masks cropping out hud elements and this helped too. In order to apply a photomask across my 400 images, I did one, exported it as a file, and then imported it as a mask to all the 400 remaining as a file with default settings. It should also be noted that I learned the hard way NOT to apply masks to key points or tie points, this resulted in an empty mesh for me. Just use None.
• I manually deleted outlier voxels from the dense cloud to help make a meaningful mesh. This Agisoft tutorial was really well done, takes some patience to get through but you really need to see all the clicks and the reasoning behind it.

Now I'm going to try and use a smaller (80 picture) dataset, but I carefully applied masks so it doesn't look at the background so much. This didn't work =(. The background must've helped It was too sparse. I tried combining my 400 and 80 images and that didn't work either. Then I started over on my 400 image dataset that had worked well, then aligned only the 1st 100 images, then aligned the remaining 300, this went fairly fast likely due to the dimensionality being cut in half. I theorized that doing so can give me a suboptimal alignment for the 1st 100 so I reset and realigned those. Then after all 400 were aligned I did an alignment optimization function which is only available in batch options.

I have some more work skill refinement on this, but now I feel much more confident in photogrammetry, by manually massaging the data. Geez! I thought it'd be click play go! I guess this is why they say its an art.

Installing Tensorflow 2.0 on Windows (NOT 1.14!)

Unfortunately, the process I wrote resulted in tensorflow 1.14. what I really wanted was tensorflow 2.0.

Googling yielded pretty much the same instructions that I executed so i'm a bit at loss:
https://medium.com/@shaolinkhoa/install-tensorflow-gpu-2-0-alpha-on-anaconda-for-windows-10-ubuntu-ced099010b21


Looking around...
I ended up trying

(tf-gpu-cuda9) F:\Projects\ML\TF2>pip install --cache-dir=/data/ --build /data/ tensorflow-gpu==2.0

It upgraded a bunch of packages, but still


ERROR: tensorflow 1.14.0 has requirement tensorboard<1 .15.0="">=1.14.0, but you'll have tensorboard 2.0.1 which is incompatible.
ERROR: tensorflow 1.14.0 has requirement tensorflow-estimator<1 .15.0rc0="">=1.14.0rc0, but you'll have tensorflow-estimator 2.0.1 which is incompatible.
ERROR: tensorboard 2.0.1 has requirement grpcio>=1.24.3, but you'll have grpcio 1.16.1 which is incompatible.

to fix this I did the following, i had to uninstall tensorflow 1.14 because it upgraded everything except tensorflow which was stuck at 1.14. If I tried to install tensorflow with a pip install it through this error:

EnvironmentError: [Errno 28] No space left on device

the solution to uninstall and reinstall tensorflow 2.0 was to do this:

pip uninstall tensorflow
pip install --cache-dir=/data/ --build /data/ tensorflow

Installing Tensorflow on windows Links

• Anaconda for windows
• Install Cuda Toolkit 10

At cmd prompt, to confirm cuda was installed type:

nvcc -V

To start running ML code: open anaconda prompt from start  and type

conda activate tf-gpu-cuda9

Simple ML program to run

To test TF 2.0, these commands should work:

import tensorflow as tf
assert tf.test.is_gpu_available()
assert tf.test.is_built_with_cuda()

Installing TensorFlow on Windows

TF1.14 with python3 on windows this worked for me by using Anaconda to setup the virtual environments and address most of the dependencies, there still were a few things needed to be added manually while in the anaconda prompt.

The instructions say this should work to get you tensorflow 2.0.0, it looks like these commands gave me a working version tensorflow 1.14, i'll need to revisit this to get TF2.0 working properly. Also, it appears my pycharm didn't work in the venv until I installed opencv with pip (for python 2).
Strangely i learned that while i wanted to use PIL for image operations, it was depracated in favor of pillow which includes all the libraries. I wanted this so i could do screen grabs from python.
https://docs.anaconda.com/anaconda/user-guide/tasks/tensorflow/

conda create -n tf-gpu-cuda9 tensorflow-gpu cudatoolkit=9.0
conda activate tf-gpu-cuda9

pip install opencv-python
pip3 install opencv-python

pip install matplotlib
pip3 install matplotlib

pip install Pillow
pip3 install Pillow

In addition to get a basic version of tensorflow working I modeled a nuka cola bottlecap for one of her broken guns in solidworks. Simplify3D is the best slicer, i love it!

https://www.thingiverse.com/thing:3955333

Learning Blender Modeling, Rigging, Texturing

Learning to Model

I chose to install Blender 2.79 because the latest version was too hard and I'm trying to follow this tutorial:

https://www.youtube.com/watch?v=DiIoWrOlIRw&list=PLFt_AvWsXl0fEx02iXR8uhDsVGhmM9Pse

• To import background images they have to be .jpg format, others = questionable
• You have to be in Orthographic view (Numpad 5) to look at background images

Key Hotkeys

• Shift+Mouse3 - to Pan (have to try a couple times
• Shift+1 or Shift+3 - to change to front or side view
• a, then b - bounding box select
• Ctrl + R - Start cut, right click selects the middle
• Tab - to get to edit mode.
• Z - wireframe mode
• x - delete
• e - extrude
• g - groups vertices behind the one you select i think
• typical 'babg' to select group of vertices and move them.

To Round out a model (from an isometric POV)

• Alt + RMB - Select Edge
• Alt + Shift + RMB - Select another edge to close loop and get surface selection
• ... add more here....

and.... i ran out of time, so i'll pick this up later.

Playing with Mapbox

I was playing around with mapbox in Unity 2019.1.5f1 and it gave me an error, after some googling i figured out a simple solution. I'm not messing around with AR, so i deleted these folders and then the project compiled. I figured this is good too because its a bunch of AR assets that I wouldn't be using anyway:

• GoogleARCore
• MapboxAR
• UnityARInterface
• UnityARKitPlugin

Another interesting note, is I tried to load mapbox into a premade scene with HDRP settings and something with the shaders was messed up. My globe came in purple, at least when i did a blank scene it rendered correctly. I didn't feel like trying to fix it, but long term might be worth the effort to get the juicy visual quality that unity can provide. 

After poking around with the tutorial I was able to get some decent looking results. I'm pleased with this result with such little effort! Not to be picky, but I noticed some of the buildings weren't identified, I presume this is loaded from some ML defined satellite database analyzed by google or mapbox with some building classifier... Interesting!  Probably good for prototyping but this wouldn't be contextually perfect. Also, the resolution does do pretty well wherever you zoom in, but its a little worse than google in terms of resolution. 

That said, scaling up the extents and zoom really provides amazing results! What a powerful tool!

Creating Jetson OS Backup

Googling around, because I wanted to create a flashable iso image of my jetson nano before I do things that potentially break the OS... This would give me a rapid recoverable way to restore my Jetson Nano instead of the 30-60 min process it takes to re-flash, reconfigure, and reinstall all the basic software tools I want to use. Flashing a premade image takes like 5 minutes-ish.

It looks easier than I expected! It looks like I don't have to install anything to create the image or flash this on linux:

https://thepihut.com/blogs/raspberry-pi-tutorials/17789160-backing-up-and-restoring-your-raspberry-pis-sd-card

Posting a new repo to github

Jotting some quick notes on how to properly push up local files and get it synced with github.com because I'm starting to do it fairly frequently and it takes me some time...

1. Adding an existing project to github using the command-line
2. When the upload fails it says,"

git push origin masterWarning: Permanently added the RSA host key for IP address '192.30.X.X' to the list of known hosts.Permission denied (publickey).fatal: Could not read from remote repository.

Please make sure you have the correct access rightsand the repository exists."

3. generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
4. adding-a-new-ssh-key-to-your-github-account
5. I messed up by creating a readme, and so now it gave me this error:

$ git push origin masterTo git@github.com:nickswimsfast/SpatialPerception.git ! [rejected]        master -> master (fetch first)error: failed to push some refs to 'git@github.com:nickswimsfast/SpatialPerception.git'hint: Updates were rejected because the remote contains work that you dohint: not have locally. This is usually caused by another repository pushinghint: to the same ref. You may want to first integrate the remote changeshint: (e.g., 'git pull ...') before pushing again.hint: See the 'Note about fast-forwards' in 'git push --help' for details.

The solution (but rough and risky solution) was to:

git push origin master --force

6. create-a-gitignore

ML Dive

Diving into some ML... While I have cloned repo's and applied them successfully with transfer learning with models like inceptionv3, and resnet, I wanted to get in a little deeper. Before I could get started, I set out to setup my VR computer (GTX 1070 Graphics) as a development box. This entailed getting dual boot windows/ubuntu setup along with VNC.

Tensorflow Setup
I was surprised because just getting my computer properly setup to use GPU accelerated machine learning was more of a chore then I expected. It took me a solid 3 hours to decipher all the things I had to do in order to get actually training neural nets on the GPU. I'm confident it was worth it for speed, but geez!!! It shouldn't be this hard! Docker, Cuda 10 drivers, linking to the OS so I could save files permanently, etc... Anyway, I documented the entire arduous process on github - a little messy because they were quick notes for myself, but it might help somebody:

[SetupML Github repository]

VNC = dev speed
Can I just mention the ability to log into my VR computer from anywhere in the house is a godsend! I can sit with my laptop and work on it from the couch, from the bed, from my desktop computer or laptop! So much more productive.

Getting started with ML
Once I was setup I verified things were working with these quick tutorials:

* [Basic Classification]
* [Text Classification]


Synthetic Data
Enough with the tutorials! I wanted to branch out on my own more or less scratch built perception model, so I crafted up some semi-automatic 200 synthetic images with truth labels in Unity and set out to work. I spent pretty much this whole weekend putting all this together getting data formatting right, arrays setup with the right datatypes, so I can have an automatically setup training and test data sets.

Complex Model
Unfortunately, I'm trying to get my last layer to be a multi-dimensional regressive output, this is proving to be more challenging than I thought. The few times i successfully trained my rough models, I was getting at best 60% accuracy. I've got to spend a few more days researching keras model architecture, I think its bad. Been messing around with the sigmoid, linear, and softmax activation functions, along with various loss functions. I stumbled onto these interesting links which might help me find the answer:
* [Keras Github Examples]
* [Keras Documentation - with examples]

Whew! Time for sleep!

First Github Repo!

I've made several github repositories on at work on our local network's git website, but I had never posted my own repo outside that ecosystem. Today is my first public repo on github! I made some scripts to make it easier to change the clockrate/power consumption of the jetson nano instead of having to remember all the commands and syntax. Here it is:

Jetson Power Scripts
[Jetson Nano power management](https://github.com/nickswimsfast/jetson_powermgmt)

Good instructions for starting a [new github repo](https://kbroman.org/github_tutorial/pages/init.html).

SSH/VNC to Jetson Nano
I setup ssh beforehand with vino on the jetson itself - the default instructions weren't exactly right, but some googling later and you'll git it!. Also, I've found that using realvnc works great for connecting to it (free solution). TightVNC costs money?! Also, on my x86 windows desktop computer it was easy to use ubuntu's built-in remote desktop sharing. I also had to install openssh-server on the x86 computer, but I don't think i needed to do that on the Jetson, i think it came with that by default.