I gonna keep this short as I can so first thing first, I gonna to list program that are available to Linux right off the bat and other program that required to use Wine to work.
(Green mean it available for Linux/ Red mean it required to use Wine)
Program you can download for Linux:
GMod (It 10$ but allow you for ragdoll and shit; Steam Required though but Valve allow their games to be supported for Linux; in fact, recommended it!)
Fraps (It is discontinued but many still uses it for recording and still up. Only on .exe files)
Vega Pro (Very Professional editor that everyone uses; Only for Windows which Wine could work but who wanna to pay $600?)
*Important info: You must have Super Mario 64 game. You can download it but you must Google it, Bing it or Duck it (whatever the fuck Search Engine you use). I can’t risk my site hosted illegal shits here (as well as Neocities getting punished on Google ranking). You can also legally import SM64 to your computer as well.
**Additional Info: It recommended to get Green Screen pack if you wanna to do SMG4-style video but it optional. There also collection of Mario characters as well as mood-set and Green Screen. Here the link to download it (Thanks WeegeePie!)
For newbies, in short, Wine (a.k.a Wine Is Not a Emulator) is Linux open-source software that allows Linux users to download and run Windows .exe software using Wine Loader. It treated like an Windows apps and in most case, this will work on any .exe files you needed. There other Windows Loader program you can use on Linux but Wine is popular choices. It is free so you can download from here.
I want to point it out that on the pack (with Green Screen and characters), you need to unzip that files in order to pick out selection you want to pick. Each file has .z64 files which will work on P64. The game files itself should not be unzip!
In short, you need Project64, an SM64 game and Video Editor (I recommended OpenShot since it Free and easy to use but Vega Pro is very professional tool, though it $600 or more). You also need to have GMod and Green Screen if you wanted to do more complex and more Modern SMG4-style but if you just wanted to do more classic style, just record SM64 and you good to go.