2019-05-29 00:14:55 (edited by amerikranian 2019-05-29 00:15:36)

So, my pyinstaller is dead set on creating exes for some reason.
If I try and compile a program that uses cython scripts, the thing goes nuts. Typing the following yields the same result, that is, an exe which does not show tracebacks unless I redirect the stderr.
1: pyinstaller start.spec #Should create a directory with a bunch of files in it with the start exe. In actuality, it creates an exe, which it shouldn't do. Furthermore, that exe behaves as if I typed --windowed, which I didn't.
2: pyinstaller start.spec --onefile #Produces an exe with the inability of seeing the tracebacks when in reality I should see them because I didn't type --windowed.
So, help? I really miss my --onefile and seeing tracebacks...
Here's what I've tried.
1: Reinstalling pyinstaller. Didn't help.
2: Checking and upgrading to the latest version of everything by using pip-review --interactive and then typing in A, didn't help either.
So, tips? Suggestions? Help me get pyinstaller back under control again, LOL.

2019-05-29 04:45:27

regarding cython, first you should compile those pyx files into pyd (by cythonizing).
then, try to compile using pyinstaller mainscript.py and then change the spec file and add everything that those pyd files need in hiddenimports, and you are good to go.

2019-05-29 05:09:18 (edited by amerikranian 2019-05-29 05:10:31)

I am aware of how Cython works.  My problem is not with that. My problem comes with pyinstaller not listening to the commands I supply to it.

2019-05-29 07:06:41

does it give you any warning during the compilation process?

Paul

2019-05-29 13:18:40

From what I could tell, it gives me the same errors as it did before, which does not really impact the application. I might have missed it though, it’s a lot of text

2019-05-29 14:00:13

what happens if you compile your code uncythonized?

Much less active on this forum than in the past.

Check out my live streams: http://lerven.me
follow me on Twitter: http://twitter.com/liamerven

2019-05-29 17:04:01 (edited by Sam_Tupy 2019-05-29 17:05:47)

I'm slightly confused. Pyinstaller's job is to create an exe. Also, try adding the arguments such as --onefile to the start of the command before the filename, like pyinstaller --onefile script.py. Also, delete the spec and use a py file instead. A spec file contains all those commands, so if you run it with the spec file with new commands I'm not entirely sure the spec file will update. Just run pyinstaller start.py instead. Another thing I found out is that windows has this extremely weird thing it does with certain filenames. If you build a file called update.exe and run it, it automatically elevates it's privilage to administrator and it's really hard to see tracebacks. It may do this with start.exe. It does it with update, setup, install, and maybe others. So yeah in short if you are passing a spec file, it makes sense that pyinstaler isn't listening to commandline args, I think the spec takes presidence. If that spec says onefile, it'll pack with onefile. If that spec says windowed, it'll pack with windowed. IT doesn't matter what your cmd args are. pyinstaller automatically creates spec files whenever you run based on the cmd args, then I always just delete them. I honestly don't get the idea behind the system, it may be a way to make a project that is always compiled the same way to have the same long list of hiddenimports and other such things so you don't need to pass a long list of args in every time you want to compile. But if you want to use the spec file and not the py file, then mod the spec file with the args you want such as windowed or onefile etc

I am a web designer, and a game developer. If you wish see me at http://www.samtupy.com

2019-05-29 18:00:44

So if you don't use .spec files, how do you tell pyinstaller what cython modules it should include?

2019-05-29 18:08:05

you use things called hooks.
Those hooks define which modules should the application import.

Paul

2019-05-29 18:49:47

Can you show me an example of using hooks instead of hidden imports?

2019-05-29 19:07:00

Hooks are basically python modules which allow you to setup hidden imports and exluded imports for your application.
They are basically used for certain modules if for example they have some dependencies to import.
Sof if you for example created a third party library which has sound_lib as a dependency you would have to create a module called hook-libraryname.py and do something like the follow.
hiddenimports=["sound_lib"]
But generally you will have to do that only if you have some modules which rely on dependencies.
You should be fine in most cases.

Paul

2019-05-29 19:23:02 (edited by amerikranian 2019-05-29 19:23:42)

I still do not understand.  If I have multiple modules, do I create a hook file for each one?  Why is using Hooks better than hidden imports?  Could you maybe create a set of sample scripts and upload them so I can take a look  at the structure ?

2019-05-29 19:56:49

most likely when you install modules, they already have an hook by default so you don't have to worry about that.
Hooks are used to manage hidden imports.
I apologize, it was my fault mentioning hooks, because I missunderstood what you were asking.
Remember this. Hooks are used by third party libraries to import their dependencies. You most likely will never have to deal with them.

Paul

2019-05-29 20:49:09

For simple cases, you can just use --hiddenimports from the commandline.
For instance
pyinstaller -F --hiddenimport=sound_lib myapp.py

2019-05-29 21:34:55

do the following:

pyinstaller script.py --onefile --windowed --noupx

then edit the spec file and change whatever you want (you can turn Console to False or True, which is the oposit of --windowed mode).
also, do not pack with upx in first hand with pyinstaller itself, since it will pack msvc runtime and it will break everything.

2019-05-29 23:16:50

Post 15, I’m not sure I fully understand you. Why would using UPX break things. When I did not supply that argument, it worked fine.

2019-05-30 00:01:35

OK, hold on, lets clear up a bit of confusion here. Now that I have more than 5 minutes to write a post at the end of a class, here we go. So paul is somewhat wrong, if you use cython, you will likely want to use a pyinstaller hook to manage your hidden imports. So, you will probably need to use them. Hooks aren't just for hiddenimports, hooks are used when you have a package that needs special treatment when compiling. Maybe it's moving a dll file to the proper place or maybe it's just hiddenimports or maybe it's printing a warning to the user saying hey bro please upgrade me to compile. Using hooks for hiddenimports though is at least what I do and it works quite well. No, you don't need one for each module. Basically what you do is choose a module, any module. It needs to exist though. This may or may not work with the main script, I've never tested that, so I'd just use a module. So lets say our module is called game.py. Just encase you need other custom hooks, what I would do is create a folder in your games running directory called hooks. Now, create a file called "hook-game.py" in your hooks folder. This is actually a python script, allowing you to import stuff and do whatever pythonic thing you need to. In this case, all you need is hiddenimports. Basically pyinstaller will exec this python script once you tell it about the hooks folder, and access the hiddenimports global of it. So say in your cythonized file, you include the modules os, threading, enemy, json, and item. To tell pyinstaller about these imports, add this to your hook-game.py file of course excluding the start and end quotes. "hiddenimports=["os", "json", "threading", "game", "item"]". Now, to tell pyinstaller about your hook, pass the additional argument --additional-hooks-dir, like this. "--additional-hooks-dir=hooks". Now when you build with that argument, the hook-game.py file will be executed and the modules from the hiddenimports variable you declared in it will be included. I have not tested it and am not sure of the sintax, but you can also declare hiddenimports through the command line. I believe there is a --hiddenimports arg or something, but as I said I never tested for how to use it. You can also use spec files like you were doing before, but as I said in my previous post, you can't change command line arguments when using a spec file as far as I know. I don't use spec files at all so others may have a much more complete knowledge of this than me. But from my understanding, spec files are actually built using your command line arguments. So when you build a spec based on your py script, like pyinstaller --onefile --windowed script.py, it will make a spec file based on those arguments. So now, to recompile, you can call pyinstaller script.spec and it will use the arguments you first used to create it. Thus, if you pass new arguments, they won't be used because the spec file overrides them. To change them, you have to modify the spec file in notepad, as visualstudio pointed out. As I said I'm not completely clear on it so take what I say about spec files with a grain of salt. However, there is a much, much less annoying approach you can take. Why bother with hiddenimports at all! Remember, all your trying to do here is to tell pyinstaller that a module you need must be imported. By default, pyinstaller scans all scripts it loads for import lines and uses that to figure out imports. So, all you need to do is in your main script that you are compiling, that launches your application, just add them all there. "import os, threading, json, game, item". Now compile, and it'll work perfectly. All you need pyinstaller to do is to import the modules. It doesn't matter what scripts use them, they just need to be in the pyinstaller package, and then any module that needs can use them. So what I'm trying to say here is that imports are separate from the modules that import them. Basically on compile, pyinstaller finds all the imports it can, and tosses them into your final package. When you run that package, it doesn't matter where they came from just that they are there. Your modules will import what's needed, and if it exists, your app will start. So knowing this, you can skip the step of hiddenimports entirely. Obviously you have to start from a python script when you compile with pyinstaller, so in that script, just make sure to keep updating it with new import lines whenever you add a new one to something that will be cythonized. So long as those imports are in a file pyinstaller can read, and of course it can read your main file, those imports will be included and your app will work just fine. Then, you don't need spec files at all. Ultimately what path you choose to take is up to you. Hope this makes things more clear, please don't hesitate to ask questions.

I am a web designer, and a game developer. If you wish see me at http://www.samtupy.com

2019-05-30 01:02:23 (edited by amerikranian 2019-05-30 01:05:50)

So, just to clarify:
Hooks are over glorified hidden imports. Their files contain the line hidden imports= module names.
Spec files are preset arguments that define your program. they ignore any arguments that are passed along to them if they are not in the spec file.
The most simplest way  to include everything is to have your main python script import all the modules you need and then start the app.
Did I miss anything?

2019-05-30 01:13:23

Hooks can be used for hidden imports, but they are able to be used for much much more, and the correct line sintax is in my post unless the forum broke it. Other than that though no you did not I believe

I am a web designer, and a game developer. If you wish see me at http://www.samtupy.com

2019-05-30 01:25:31

So slightly off topic, when I compile I usually have two versions of the script. I have the .py version, and I have the .pyd.  What makes the program include the .pyd and not the .py? This is just out of curiosity.

2019-05-30 03:21:11

I believe there is some sort of optimization that is performed where the faster things are loaded before the slower things. It follows the same python import order. First it checks for C extensions, because those would be the quickest. Then, it checks for pyc files both in pycache and out, because they would be the next quickest. Then, it checks for module.py, or a folder called module, because those would be slowest as they would have to be compiled into pycs, or if not their code would at least have to be parsed, and string parsing is always the slowest. So in short, pyinstaller follows the same loading order in general as python's own import system, *.pyd/.so if not on windows, *.pyc, *.py

I am a web designer, and a game developer. If you wish see me at http://www.samtupy.com

2019-05-30 03:28:22

Thank you. I believe you have exhausted my question supply for now.