Skip to content

Should scikit-build pick the right compiler on Windows automatically? #668

Closed

Description

@ChrisBarker-NOAA

We've recently ported a pretty complex project with C, C++ and Cython used as Python extensions to sikit-build-core.

We got it all working on Mac an Linux, but on Windows, we're having some issues.

(Using Powershell: it seems to be required by our CI)

Our first attempt resulted in it building for x86 instead of x64 -- it turns out that we had (carelessly ) used a Powershell configured for x86. And we're making progress with a generic powershell and activating the x64 compiler with: vcvars64.bat.

However -- shouldn't scikit-build (or cmake) figure that out for us, and call the right compiler, based on the Python that's doing the build? I'm a bit concerned that setting it up by hand ahead of time is a bit fragile for less experienced uses and CI systems, etc.

Or is that just that way it is?

Note: this could be related to #590 -- though that made me think that it should "just work"

-CHB

Activity

  1. LecrisUT commented on Mar 19, 2024

    @LecrisUT
    Collaborator

    Do you use a GHAction like ilammy/msvc-dev-cmd@v1? What issues do you see, is it in the compiler chosen or in the build artifacts produced?

  2. ChrisBarker-NOAA commented on Mar 19, 2024

    @ChrisBarker-NOAA
    Author

    we're building locally at the moment, and will be using our in-house gitLab CI (and ultimately conda-forge, but baby steps...)

    It's the compiler (or how it's set up) -- it seems to simply use the default set up in the shell -- so it was trying to build 32 bit extensions agains 64 bit libraries.

    Never got as far as build artifacts :-)

  3. LecrisUT commented on Mar 19, 2024

    @LecrisUT
    Collaborator

    There are various factors that influence the compiler detection, but generally it's environment variable CC and what resolves cc in PATH (similar for CXX). For window it should resolve to cl.exe. But about the 32 vs 64 bit issue on Windows, I am not familiar enough to debug that. From what I've read on ilammy/msvc-dev-cmd@v1 it basically calls vcvars64.bat so that cl.exe to the appropriate value. It did specify this nuance iirc, but I couldn't follow it. Maybe you can decipher it better.

    As for conda-forge, it works automagically, on this issue.

  4. ChrisBarker-NOAA commented on Mar 19, 2024

    @ChrisBarker-NOAA
    Author

    it basically calls vcvars64.bat so that cl.exe to the appropriate value.

    Ahh -- that's what I thought, but it didn't seem to be happening in our case.

    From what I've read on ilammy/msvc-dev-cmd@v1

    What's that? where in code would that be?

    -Thanks

  5. LecrisUT commented on Mar 19, 2024

    @LecrisUT
    Collaborator
  6. bennyrowland commented on Mar 20, 2024

    @bennyrowland
    Collaborator

    This is purely anecdotal, but I am currently going through a conversion of a 32-bit project to 64-bit, on Windows, and I was very pleased to find that scikit-build-core did "just work" for me. I use tox and have two environments set up (py39-32 and py312-64) that use different base_python values for 32 and 64-bit respectively (the same is true for their packaging environments as well). Each one correctly identifies the right toolchain to compile the extensions to the right bit-ness, without any further input from me. That is running on a standard powershell without pre-setting up any compilers or anything - is it possible that when you activate vcvars in advance you prevent scikit-build-core from doing its own activation?

  7. LecrisUT commented on Mar 20, 2024

    @LecrisUT
    Collaborator

    Afaik, there is no secret sauce for detecting compilers within scikit-build-core, and everything is based on how CMake detects it. There is a bit of manipulation for the -G generator, but other than that, it should be as close to vanilla CMake as possible, with only additional SKBUILD cache variables. Hopefully I got this right @henryiii.

    So, if you are able to get your CI working for plain CMake, adding scikit-build-core should be straightforward. Although, I personally always add the CMAKE_C_COMPILER option explicitly in the CI.

  8. henryiii commented on Mar 20, 2024

    @henryiii
    Collaborator

    We do want to get the same compiler setuptools would get if things is specified. If you set up your shell with x86 (by clicking on the "x86 command line" shortcut or running vcvars.bat) but use a 64-bit Python, that's asking for trouble. But if you don't activate anything, it should ask CMake to find MSVC with the correct bittiness. You shouldn't need to manually run vcvars (such as through ilammy/msvc-dev-cmd) for most cases. We try to match Python's bittiness unless it's specified (VSCMD_ARG_TGT_ARCH, which I think vcvars sets) here:

    def get_host_platform() -> str:
    """
    Return a string that identifies the current platform. This mimics
    setuptools get_host_platform (without 3.8 aix compat).
    """
    if sys.version_info < (3, 8) and os.name == "nt":
    if "(arm)" in sys.version.lower():
    return "win-arm32"
    if "(arm64)" in sys.version.lower():
    return "win-arm64"
    return sysconfig.get_platform()
    def get_platform(env: Mapping[str, str] | None = None) -> str:
    """
    Return the Python platform name for a platform, respecting VSCMD_ARG_TGT_ARCH.
    """
    if env is None:
    env = os.environ
    if sysconfig.get_platform().startswith("win"):
    if "VSCMD_ARG_TGT_ARCH" in env:
    logger.debug(
    "Selecting {} or {} due to VSCMD_ARG_TARGET_ARCH",
    TARGET_TO_PLAT.get(env["VSCMD_ARG_TGT_ARCH"]),
    get_host_platform(),
    )
    return TARGET_TO_PLAT.get(env["VSCMD_ARG_TGT_ARCH"]) or get_host_platform()
    if "arm64" in env.get("SETUPTOOLS_EXT_SUFFIX", "").lower():
    logger.debug("Windows ARM targeted via SETUPTOOLS_EXT_SUFFIX")
    return "win-arm64"
    return get_host_platform()

  9. ChrisBarker-NOAA commented on Mar 20, 2024

    @ChrisBarker-NOAA
    Author

    @henryiii: Thanks! That was what I thought should happen, apparently we were trying to be too clever.

    I'll report back if/when we get it working, and we can then close this.

  10. bennyrowland commented on Mar 22, 2024

    @bennyrowland
    Collaborator

    Ok, I might have been premature when I said that this was working for me - I have now found a really confusing situation. To test it, I have created a dummy project which contains only an empty CMakeLists.txt and the following pyproject.toml:

    [project]
    name = "test"
    version = "0.0.1"
    
    [build-system]
    requires = ["scikit-build-core[pyproject]>=0.8.2"]
    build-backend = "scikit_build_core.build"
    
    [tool.tox]
    legacy_tox_ini = """
    [tox]
    envlist = py312-64
    
    [testenv]
    base_python = python312-64
    package = wheel
    """
    

    I currently have a 32-bit Python 3.9 and a 64-bit Python 3.12 installed on my system (and VS2019). I have created new venvs from both 3.9-32 and 3.12-64 and run only pip install tox on each of them, then run tox from each environment on the test project. The 3.9-32 environment succeeds in building the wheel correctly (using a 3.12 virtualenv) while the 3.12-64 environment fails with output of

    -- Building for: Visual Studio 16 2019
    -- Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.22631.
    -- The C compiler identification is unknown
    -- The CXX compiler identification is unknown
    
    

    If I manually run cmake.exe from the 3.12-64 tox's packaging virtualenv then it correctly finds the compiler without issue, and if I run either python -m pip wheel . or python -m build from either env then I get the wheel I want. So I cannot figure out what could be going on within tox that could be causing this issue. @henryiii, I will raise this issue with tox as well, but do you have any insights into what could be going wrong that might be causing this issue?

  11. henryiii commented on Mar 22, 2024

    @henryiii
    Collaborator

    Tox isolates you from the surrounding environment variables by default. Nox does not, by the way. You can set a tox config to pass through environment variables. That’s my first suspicion for what might be surprising.

  12. bennyrowland commented on Mar 26, 2024

    @bennyrowland
    Collaborator

    @henryiii good shout to check the environment variables, thanks - it turns out that the magic one that needed to be passed through was WINDIR. I still haven't been able to figure out exactly why that is important to pass to 64-bit Python but not to 32-bit - it is related to the PROCESSOR_ARCHITECTURE environment variable. I think that as far as CMake is concerned, if you call it from 32-bit Python then it thinks it is running on a 32-bit system and basically cross-compiling for 64-bit Windows via the generator architecture flag scikit-build-core passes it, so it can find Visual Studio in the right location, whereas from 64-bit Python it somehow needs WINDIR in order to track down the (32-bit) Visual Studio installation. Maybe this will change with VS2022 which is 64-bit (and might just break running tox from 32-bit Python instead).

    Interestingly enough, I have been testing out using Nox as a possible replacement for Tox, but for a different reason. I am working on getting another of your amazing projects, cibuildwheel set up to do my wheel building, at least for my CI, and I want to reuse (at least one of) the built wheels to build the docs and any other post-test jobs that need doing. My idea was telling Nox (as a very thin wrapper around pip) to install the package from the wheelhouse folder rather than building it anew. Tox supports passing in an external package, but I don't want to have to deduce the name of the actual wheel file for whatever version of Python is currently building my docs, just assume that cibuildwheel has built one that matches and let pip go through the wheelhouse to find the one it needs, which Nox allows me to do very easily. I am still trying to work out whether this is a good idea, or something I am going to regret in 6 months...

  13. ChrisBarker-NOAA commented on May 30, 2024

    @ChrisBarker-NOAA
    Author

    @JamesMakela-NOAA: I think you solved this -- if so, maybe post what worked for you, and then we can close, if no one else has ongoing questions.

  14. henryiii commented on Jun 29, 2026

    @henryiii
    Collaborator

    I think this is a bit better on main now, and quite a bit more flexible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions