A better GUI

[Xymanek]

Alright, soooooo..... One of main things that I was expecting to see when I've found this project is a GUI matching the one in the show. Machine 1.0 sort of accomplished this (aside from camera being in a separate window). Machine 2.0 is (no offense to creators, great job anyway) a joke in this aspect. So after thinking about this for quite some time I came up with a list of ideas/suggestions how to make the GUI more matching to the real (well, original) one. Sadly my java skills are very low (I'm a web dev, work with PHP, HTML, CSS, JS) thus I cannot attempt to implement my ideas myself.

• Revert the main window to the way it was in v1.0
• Get rid of the pop-ups (such as the camera or the terminal)
• Change the "information" box. The one from the title sequence doesn't look good unless it's on top of a full screen image (it simply gets cut off). A better one might be something like this.
  
• At the point where it would completely load (saying "admin detected..." or "secondary detected...") replace that with "Main screen"

What do I mean by "Main screen":
This is a "replica" of what we see most of time in the show (not the full screen camera feed). Examples:


(without the "contextualizing" text)

• Place the webcam "feed" in the center (so it can remain at 1:1 ratio)
• If multi-input support is added those feeds can placed on sides (left and right), user can scroll through them by pressing left and right on their keyboard
• If the webcam resolution is small, the information box can be placed outside of image and linked with a yellow/white/red (depending on classification) line to the colored box around the face
• The speech recognition can be placed on top of the main image in a similar manner to when machine is analyzing audio feed in the show. Example (notice that the text isn't stationary, it is scrolling right to left):
  
• This can be used as a background (or maybe just some static blue color):
  
• Under the camera (inside the "main screen") we can place the terminal (which can optionally slide down and up)

This is a rough draft of my ideas which I'm putting here up for a discussion. I know some people might disagree, some might say that they are too hard to code or something similar. I'm just interested to see if there are other people out there who share my opinion.

Also if this is accepted this is probably gonna be another long-term issue as such gigantic redesign will obviously take time. Maybe use another branch for it?

[devloop0]

Some thoughts on your issues:

1. You are right about the Machine 1.0's UI, it probably looked a little cleaner than what the Machine looked like now, although I feel the startup animations are probably a little better. Window decoration-wise, it is definitely a step down. I feel that forcing the machine fullscreen might be a better option. Window resizing with this many operations may be too difficult to handle with so many operations running in the background. This is something that I will definitely try to work on.
2. If I fullscreen the machine, then the camera popup can definitely go, but I'm still not convinced on the terminal popup. Since we are trying to emulate the show, in it, Harold clearly opens popup Windows with the terminal, and it might make sense to do the same with our machine. An alternative could be that when the user requests to open the Machine's terminal, one half of the screen could be dedicated to a stream while the other half would be dedicated to the terminal.
3. This is an aesthetics problem that has been bugging me for some time. I see your point, and I want to stick to the current look (the title UI's flat look I feel is superior to the one shown), although I am open to hearing counterpoints on this. But you are right, in that the positioning is always off. The problem with putting it where you have shown it is that information would still get cut off. If you notice your first screen shot, John is still being displayed on a full monitor (near the edges of the screen). How would you/anyone proceed with this?
4. The main screen is probably a good idea. As I said before, a better target UI is probably a fullscreen application with a close and minimize button (to avoid handling resizing) (and by full screen, I mean like games, where even the task bar goes away). Additionally, your idea of choosing an IP cam is definitely feasible and definitely doable. This is another reason why I want the terminal in a separate window. In our current state, with separate windows, the size of the camera is completely separate from the size of the terminal, combining them into one screen forces more logic that I feel does not necessarily add more usability or aesthetics. I am also open to hearing counterpoints on this. I know that in Windows 10 (not sure about older Windows versions), it is possible to run windowed windows (like the Machine terminal), over applications in full-screen (like pressing Win+R while playing a full-screen Youtube video). Additionally, a potential problem that I could see is what happens when you insert an IPcam or a regular webcam that takes up the entire screen. In our current mode, the size of the webcam is irrelevant, but if we combine the terminal and the webcam, then deal with this becomes more difficult. Your suggestion of "under the camera" could also not work with larger webcams and additionally, all the issues with a side-by-side display are there.
5. I like the idea of switching between multi-input modes with arrow keys. Additionally, if the machine currently cannot find a webcam, it simply exits, but maybe adding a no-signal message like r/personofinterest shows with spoilers (static with no signal written on top), might be a better idea.
6. If you notice in your screenshot for speech recognition, there is no webcam. If you have one with a webcam, I would like to see it. The problem is that facial detection runs every second, facial recognition runs twice a second, and speech recognition runs only when someone talks and the recognition can take an arbitrarily long amount of time. Syncing the first two was not difficult, but syncing the last one with the first two is significantly more difficult. Additionally, the scrolling effect you see is because what the machine was listening to was a recording; I guess the best thing we can do now to compensate is to just scroll until all of the text is invisible and then stop scrolling. Thoughts?
7. Placing the webcam recognition info beneath with small Windows would result in a substandard experience. I agree that the current experience is substandard, the problem is that most webcams are too small to support the info that we are displaying in the way we are displaying it. I'm conflicted about this as well because, referencing your first point, no matter where we place the Machine box, it will always be cut off. Maybe performing facial recognition and placing numbers next to the detected face in the webcam and referencing them beneath via a numbered-list might be a better option.
8. I'm not sure how applicable running a machine instance from a video file is. The feature was supported in the old machine and can be added in the new one, but I'm not sure how applicable it is.

I feel that this probably addressed all of the issues that you had. (Just remind me beneath if I forgot one).

Now onto implementation stuff:

1. If we keep the terminal separate from Machine, then the only issue remaining is to make the application full-screen, add the appropriate "Window-decorations," recenter the Machine webcam Window, and support all the given webcams by making it scrollable. It definitely adds complexity to the application, but this definitely seems doable.
   If we integrate the terminal into the window, then in addition to all of this, we will also need to keep track of the webcam size. I would definitely like to hear other thoughts on doing this, as this will add significant complexity. (Additionally, since this will support multiple webcams, each time there is a switch, everything will have to be re-rendered). I think if we want to support this, and the webcam is bigger than half of the screen's with (in a side-by-side scenario), then maybe making the terminal's background transparent is a good compromise solution). In this case, since the window size is always fixed, this should be doable, but more difficult.
2. Speech recognition will still be above the webcam output, but with scrolling as well.
3. The sliding terminal window idea that you suggested could also be done as well, although that will be a little more complex as well.

Some final details:

1. I do not have access to an IP camera, which means that I cannot test such a feature. As such, I will not implement it, as I do not like releasing code that I cannot test (unlike Harold and Root, not everyone can code bug-free code without testing it). If there is a public IP camera that I can access, a URL would be nice.
2. I would like to use images from the show, but I am not sure whether they would like images to be re-distributed so freely (everyone who downloads a binary copy of the machine will get an image).

As a side note, everything I have just said could be thrown out and people could easily say that they prefer the current version of the machine because it is more flexible (i.e. things can be moved around and you can switch between windows easily), but I feel that there is some truth in what Xymanek is saying, in that the current looks is somewhat lacking. A fullscreen machine window would definitely look interesting though but it is clearly up to the community. If more people feel the way that Xymanek feels, then I would be glad to try my best to implement what I have described in my post.

Feel free to discuss.

[Xymanek]

I feel that forcing the machine full screen might be a better option

Completely agreed! To be honest that's what I thought from the beginning but since both versions were windowed I thought that you decided that it shouldn't be full screen

So this is what the layout should look like in my opinion (sorry for poor quality, this took me 1 min in MS paint):

Harold clearly opens popup windows with the terminal, and it might make sense to do the same with our machine

That's what I meant but instead of using a OS's window, we make one ourselves which is bound to bottom of the screen (so the whole machine is together in one window)

Your suggestion of "under the camera" could also not work with larger webcams

If we make the terminal window to slide up/down I think it's ok if it covers a small part of video feed. Additionally we can make the feed frame in two modes: preview (small, where we have left and right feeds in row) and focus (where it's zoomed in). Thus the top-to-bottom of layers is:

1. Terminal
2. Speech recognition
3. Focused feed
4. List of feeds
5. Background

Additionally it is possible to make windows move around based on where the identified faces are

[Xymanek]

Second part of my answer (yesterday was too busy):

This is an aesthetics problem that has been bugging me for some time. I see your point, and I want to stick to the current look (the title UI's flat look I feel is superior to the one shown)

I disagree with you on title > actual UI. For several reasons:

1. Title' UI was always different to the one the machine uses in the show thus it was designed for and only for the title sequence and the goal of this project is to remake the Machine from the episodes, not hand-crafted title sequence
2. In the title sequence a huge amount of text is actually unreadable (designed to be a simple placeholder) while in the actual UI all of text is readable (save for when machine was infected with a virus). However we need to actually use that text space to display data
3. The real box can be anywhere on the screen (including being partially on top of image, partially not) while the title's has to be next to the face to make sense and will look ugly if placed partially off-image
4. [Personal opinion] The show's boxes look much nicer to me and I detest the UI in title sequence (aside from season 2 and maybe 3)

The problem with putting it where you have shown it is that information would still get cut off. If you notice your first screen shot, John is still being displayed on a full monitor (near the edges of the screen)

Like I said above, a solid background gives the huge advantage of allowing to place the box anywhere. Yes it might not look that good but It would be still better than the current look. Plus a smaller size of the box gives a much higher chance to fit inside the image

just scroll until all of the text is invisible and then stop scrolling. Thoughts?

Exactly that. Store any recognized speech in a buffer and "unload" the buffer into the scrolling animation.

Maybe performing facial recognition and placing numbers next to the detected face in the webcam and referencing them beneath via a numbered-list might be a better option

Eh... No? I think this breaks the whole feel of the AI. Much better is to just let boxes jump around (where there is empty space) and connect to faces with lines.

I'm not sure how applicable running a machine instance from a video file is. The feature was supported in the old machine and can be added in the new one, but I'm not sure how applicable it is

Not run. Open as additional data feed. To simulate Machine's archive digging

each time there is a switch, everything will have to be re-rendered

I'm not good at graphical design but... Why do you have to re-render everything if you have 3 pictures in row and you just move them

maybe making the terminal's background transparent is a good compromise solution

No, that would just look weird. Just make it hide/show and when it's shown, it should have solid background to avoid any confusion

I do not have access to an IP camera, which means that I cannot test such a feature. As such, I will not implement it

In the other issue I've posted two links on how to stream using VLC. I suggest that you test the feature using that (or even get some video/audio stream reader/receiver abstraction library)

unlike Harold and Root, not everyone can code bug-free code without testing it

Um... Where did you find that? It took Harold like 5 years to make The Machine during which he was very closely supervising it. And no offense but I don't remember Root producing any software at all. All she did was hack stuff.

I would like to use images from the show, but I am not sure whether they would like images to be re-distributed so freely

Well it isn't redistribution if it's remade from scratch (and I'm not aware of any copyright rights on Machine's UI's elements). If you are referring to background idea, I think it was a bad one. A non-animated Machine's world would look really weird (still not set on this myself to be honest).

I feel that this probably addressed all of the issues that you had. (Just remind me beneath if I forgot one)

Ye I feel like we pretty much covered what each other said 😄

[apottr]

https://gist.github.com/kuanb/dbe19ce4e8ef317ee3fc - NYC Traffic camera feeds: 1 image per second, refresh the page to get a new image.
https://gist.github.com/alexwpotter/e222e644018e696ba8b60d35d285776a - MD Traffic camera feeds: the actual stream URL is in the HTML of the webpage but it's not terribly difficult to extract it. They're RTMP streams.

[devloop0]

Thanks @alexwpotter I will take a look at this.
@Xymanek: I think that the quick sketch you wrote is a good idea. I will make the terminal window semi-transparent, so even if the webcam view is huge, it can still be viewed. Additionally, if the machine will have a view like that, everything will not have to be re-rendered, as every single time an image is received from any webcam, it will automatically process everything in separate threads. Obviously, this will require much more resources depending on how many webcams are used, but considering that users should be able to open and close webcams at will, this should be fine. Opening from files can be supported as well, it's not really a hassle to add, and if it adds functionality, I would be okay with adding them.

Regarding the show's material, I was referring to the background.
As for Harold and Root, Harold coded a piggy-back onto the virus that Decima uploaded to the machine. I don't think he could have tested that, considering he didn't have access to source code of virus (if I remember correctly). Root didn't develop as much software as Harold, but considering in Season 5, she was able to code a backdoor into the machine where it was able to defend itself only if Harold said so, I think she wasn't able to test that either.
Since I have access to IP camera's I will be more than glad to try to implement this feature.
Let me know if there is anything else that I missed.

[devloop0]

The latest commit begins to address some of the issues presented here. (d51082b)

[Xymanek]

Hey! I know it's been quite long time since the commit was pushed but holidays...

Anyway, great job! One thing that I noticed is that the terminal is full width of the screen while I suggested it to be only in center. I think making it more compact makes it nicer. I'll also suggest an idea for background on gitter

Good luck and let us know when you have any further progress

[devloop0]

I was referring to this when I did that: https://cloud.githubusercontent.com/assets/2865341/15801401/08cf5574-2a9c-11e6-96a5-e7d71620a3b9.png, although center probably does look better. I will try moving it there and see what happens.