You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Should scikit-build pick the right compiler on Windows automatically? #668
We've recently ported a pretty complex project with C, C++ and Cython used as Python extensions to sikit-build-core.
We got it all working on Mac an Linux, but on Windows, we're having some issues.
(Using Powershell: it seems to be required by our CI)
Our first attempt resulted in it building for x86 instead of x64 -- it turns out that we had (carelessly ) used a Powershell configured for x86. And we're making progress with a generic powershell and activating the x64 compiler with: vcvars64.bat.
However -- shouldn't scikit-build (or cmake) figure that out for us, and call the right compiler, based on the Python that's doing the build? I'm a bit concerned that setting it up by hand ahead of time is a bit fragile for less experienced uses and CI systems, etc.
Or is that just that way it is?
Note: this could be related to #590 -- though that made me think that it should "just work"
we're building locally at the moment, and will be using our in-house gitLab CI (and ultimately conda-forge, but baby steps...)
It's the compiler (or how it's set up) -- it seems to simply use the default set up in the shell -- so it was trying to build 32 bit extensions agains 64 bit libraries.
There are various factors that influence the compiler detection, but generally it's environment variable CC and what resolves cc in PATH (similar for CXX). For window it should resolve to cl.exe. But about the 32 vs 64 bit issue on Windows, I am not familiar enough to debug that. From what I've read on ilammy/msvc-dev-cmd@v1 it basically calls vcvars64.bat so that cl.exe to the appropriate value. It did specify this nuance iirc, but I couldn't follow it. Maybe you can decipher it better.
As for conda-forge, it works automagically, on this issue.
This is purely anecdotal, but I am currently going through a conversion of a 32-bit project to 64-bit, on Windows, and I was very pleased to find that scikit-build-core did "just work" for me. I use tox and have two environments set up (py39-32 and py312-64) that use different base_python values for 32 and 64-bit respectively (the same is true for their packaging environments as well). Each one correctly identifies the right toolchain to compile the extensions to the right bit-ness, without any further input from me. That is running on a standard powershell without pre-setting up any compilers or anything - is it possible that when you activate vcvars in advance you prevent scikit-build-core from doing its own activation?
Afaik, there is no secret sauce for detecting compilers within scikit-build-core, and everything is based on how CMake detects it. There is a bit of manipulation for the -G generator, but other than that, it should be as close to vanilla CMake as possible, with only additional SKBUILD cache variables. Hopefully I got this right @henryiii.
So, if you are able to get your CI working for plain CMake, adding scikit-build-core should be straightforward. Although, I personally always add the CMAKE_C_COMPILER option explicitly in the CI.
We do want to get the same compiler setuptools would get if things is specified. If you set up your shell with x86 (by clicking on the "x86 command line" shortcut or running vcvars.bat) but use a 64-bit Python, that's asking for trouble. But if you don't activate anything, it should ask CMake to find MSVC with the correct bittiness. You shouldn't need to manually run vcvars (such as through ilammy/msvc-dev-cmd) for most cases. We try to match Python's bittiness unless it's specified (VSCMD_ARG_TGT_ARCH, which I think vcvars sets) here:
Ok, I might have been premature when I said that this was working for me - I have now found a really confusing situation. To test it, I have created a dummy project which contains only an empty CMakeLists.txt and the following pyproject.toml:
I currently have a 32-bit Python 3.9 and a 64-bit Python 3.12 installed on my system (and VS2019). I have created new venvs from both 3.9-32 and 3.12-64 and run only pip install tox on each of them, then run tox from each environment on the test project. The 3.9-32 environment succeeds in building the wheel correctly (using a 3.12 virtualenv) while the 3.12-64 environment fails with output of
-- Building for: Visual Studio 16 2019
-- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.22631.
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
If I manually run cmake.exe from the 3.12-64 tox's packaging virtualenv then it correctly finds the compiler without issue, and if I run either python -m pip wheel . or python -m build from either env then I get the wheel I want. So I cannot figure out what could be going on within tox that could be causing this issue. @henryiii, I will raise this issue with tox as well, but do you have any insights into what could be going wrong that might be causing this issue?
Tox isolates you from the surrounding environment variables by default. Nox does not, by the way. You can set a tox config to pass through environment variables. That’s my first suspicion for what might be surprising.
@henryiii good shout to check the environment variables, thanks - it turns out that the magic one that needed to be passed through was WINDIR. I still haven't been able to figure out exactly why that is important to pass to 64-bit Python but not to 32-bit - it is related to the PROCESSOR_ARCHITECTURE environment variable. I think that as far as CMake is concerned, if you call it from 32-bit Python then it thinks it is running on a 32-bit system and basically cross-compiling for 64-bit Windows via the generator architecture flag scikit-build-core passes it, so it can find Visual Studio in the right location, whereas from 64-bit Python it somehow needs WINDIR in order to track down the (32-bit) Visual Studio installation. Maybe this will change with VS2022 which is 64-bit (and might just break running tox from 32-bit Python instead).
Interestingly enough, I have been testing out using Nox as a possible replacement for Tox, but for a different reason. I am working on getting another of your amazing projects, cibuildwheel set up to do my wheel building, at least for my CI, and I want to reuse (at least one of) the built wheels to build the docs and any other post-test jobs that need doing. My idea was telling Nox (as a very thin wrapper around pip) to install the package from the wheelhouse folder rather than building it anew. Tox supports passing in an external package, but I don't want to have to deduce the name of the actual wheel file for whatever version of Python is currently building my docs, just assume that cibuildwheel has built one that matches and let pip go through the wheelhouse to find the one it needs, which Nox allows me to do very easily. I am still trying to work out whether this is a good idea, or something I am going to regret in 6 months...
Activity
LecrisUT commentedon Mar 19, 2024
Do you use a GHAction like
ilammy/msvc-dev-cmd@v1? What issues do you see, is it in the compiler chosen or in the build artifacts produced?ChrisBarker-NOAA commentedon Mar 19, 2024
we're building locally at the moment, and will be using our in-house gitLab CI (and ultimately conda-forge, but baby steps...)
It's the compiler (or how it's set up) -- it seems to simply use the default set up in the shell -- so it was trying to build 32 bit extensions agains 64 bit libraries.
Never got as far as build artifacts :-)
LecrisUT commentedon Mar 19, 2024
There are various factors that influence the compiler detection, but generally it's environment variable
CCand what resolvesccinPATH(similar forCXX). For window it should resolve tocl.exe. But about the 32 vs 64 bit issue on Windows, I am not familiar enough to debug that. From what I've read onilammy/msvc-dev-cmd@v1it basically callsvcvars64.batso thatcl.exeto the appropriate value. It did specify this nuance iirc, but I couldn't follow it. Maybe you can decipher it better.As for conda-forge, it works automagically, on this issue.
ChrisBarker-NOAA commentedon Mar 19, 2024
Ahh -- that's what I thought, but it didn't seem to be happening in our case.
What's that? where in code would that be?
-Thanks
LecrisUT commentedon Mar 19, 2024
https://github.com/ilammy/msvc-dev-cmd
bennyrowland commentedon Mar 20, 2024
This is purely anecdotal, but I am currently going through a conversion of a 32-bit project to 64-bit, on Windows, and I was very pleased to find that scikit-build-core did "just work" for me. I use tox and have two environments set up (py39-32 and py312-64) that use different
base_pythonvalues for 32 and 64-bit respectively (the same is true for their packaging environments as well). Each one correctly identifies the right toolchain to compile the extensions to the right bit-ness, without any further input from me. That is running on a standard powershell without pre-setting up any compilers or anything - is it possible that when you activate vcvars in advance you prevent scikit-build-core from doing its own activation?LecrisUT commentedon Mar 20, 2024
Afaik, there is no secret sauce for detecting compilers within
scikit-build-core, and everything is based on how CMake detects it. There is a bit of manipulation for the-Ggenerator, but other than that, it should be as close to vanilla CMake as possible, with only additionalSKBUILDcache variables. Hopefully I got this right @henryiii.So, if you are able to get your CI working for plain CMake, adding
scikit-build-coreshould be straightforward. Although, I personally always add theCMAKE_C_COMPILERoption explicitly in the CI.henryiii commentedon Mar 20, 2024
We do want to get the same compiler setuptools would get if things is specified. If you set up your shell with x86 (by clicking on the "x86 command line" shortcut or running vcvars.bat) but use a 64-bit Python, that's asking for trouble. But if you don't activate anything, it should ask CMake to find MSVC with the correct bittiness. You shouldn't need to manually run vcvars (such as through
ilammy/msvc-dev-cmd) for most cases. We try to match Python's bittiness unless it's specified (VSCMD_ARG_TGT_ARCH, which I think vcvars sets) here:scikit-build-core/src/scikit_build_core/builder/sysconfig.py
Lines 102 to 134 in 3594f33
ChrisBarker-NOAA commentedon Mar 20, 2024
@henryiii: Thanks! That was what I thought should happen, apparently we were trying to be too clever.
I'll report back if/when we get it working, and we can then close this.
bennyrowland commentedon Mar 22, 2024
Ok, I might have been premature when I said that this was working for me - I have now found a really confusing situation. To test it, I have created a dummy project which contains only an empty
CMakeLists.txtand the followingpyproject.toml:I currently have a 32-bit Python 3.9 and a 64-bit Python 3.12 installed on my system (and VS2019). I have created new venvs from both 3.9-32 and 3.12-64 and run only
pip install toxon each of them, then run tox from each environment on the test project. The 3.9-32 environment succeeds in building the wheel correctly (using a 3.12 virtualenv) while the 3.12-64 environment fails with output ofIf I manually run cmake.exe from the 3.12-64 tox's packaging virtualenv then it correctly finds the compiler without issue, and if I run either
python -m pip wheel .orpython -m buildfrom either env then I get the wheel I want. So I cannot figure out what could be going on within tox that could be causing this issue. @henryiii, I will raise this issue with tox as well, but do you have any insights into what could be going wrong that might be causing this issue?henryiii commentedon Mar 22, 2024
Tox isolates you from the surrounding environment variables by default. Nox does not, by the way. You can set a tox config to pass through environment variables. That’s my first suspicion for what might be surprising.
bennyrowland commentedon Mar 26, 2024
@henryiii good shout to check the environment variables, thanks - it turns out that the magic one that needed to be passed through was WINDIR. I still haven't been able to figure out exactly why that is important to pass to 64-bit Python but not to 32-bit - it is related to the PROCESSOR_ARCHITECTURE environment variable. I think that as far as CMake is concerned, if you call it from 32-bit Python then it thinks it is running on a 32-bit system and basically cross-compiling for 64-bit Windows via the generator architecture flag scikit-build-core passes it, so it can find Visual Studio in the right location, whereas from 64-bit Python it somehow needs WINDIR in order to track down the (32-bit) Visual Studio installation. Maybe this will change with VS2022 which is 64-bit (and might just break running tox from 32-bit Python instead).
Interestingly enough, I have been testing out using Nox as a possible replacement for Tox, but for a different reason. I am working on getting another of your amazing projects, cibuildwheel set up to do my wheel building, at least for my CI, and I want to reuse (at least one of) the built wheels to build the docs and any other post-test jobs that need doing. My idea was telling Nox (as a very thin wrapper around pip) to install the package from the
wheelhousefolder rather than building it anew. Tox supports passing in an external package, but I don't want to have to deduce the name of the actual wheel file for whatever version of Python is currently building my docs, just assume that cibuildwheel has built one that matches and let pip go through the wheelhouse to find the one it needs, which Nox allows me to do very easily. I am still trying to work out whether this is a good idea, or something I am going to regret in 6 months...ChrisBarker-NOAA commentedon May 30, 2024
@JamesMakela-NOAA: I think you solved this -- if so, maybe post what worked for you, and then we can close, if no one else has ongoing questions.
henryiii commentedon Jun 29, 2026
I think this is a bit better on main now, and quite a bit more flexible.