[FEATURE TRANSFER] Transfer MC-PDFT to the PySCF core modules

[MatthewRHermes]

Attempting to roughly follow #135

---

Authors

Matthew R Hermes (University of Chicago)
Dayou Zhang (University of Minnesota)
Thais R Scott (University of Chicago)
Matthew R Hennefarth (University of Chicago)
Bhavnesh Jangid (University of Chicago)
Shirong Wang (Fudan University)
Frédéric Chapoton (University of Strasbourg)

Feature Description

I propose to transfer the entire multiconfiguration pair-density functional theory (MC-PDFT) implementation from PySCF-Forge to the PySCF core modules, excluding dipole-moment features for which the corresponding implementations in other quantum chemistry methods are not currently hosted in the PySCF core modules. MC-PDFT refers to methods in which the total electronic energy is obtained or derived from a functional of the on-top pair density as well as the total electron density, and these densities are in turn obtained from a multiconfigurational self-consistent field (MCSCF) wave function or wave functions of some sort. The on-top pair density is the probability of two electrons existing simultaneously at one point in space.

The non-classical part of the MC-PDFT energy is called the "on-top energy" and is directly analogous to the exchange-correlation energy in standard density functional theory. In practice, the on-top energy functional is always a generalization or a "translation" of an exchange-correlation functional from Kohn-Sham DFT. MC-PDFT furnishes a practical and effective generalization of KS-DFT functionals from single Slater determinants to general multiconfigurational wave functions because the pair density, unlike the spin density, can simultaneously encode both strong electron correlation effects and spin symmetry constraints.

This feature implements theoretical developments discussed in the following publications:

• Original MC-PDFT publication: JCTC 2014, 10, 3669
• Functional development publications:
  • Fully-translated LDA and GGA on-top functionals: JCTC 2015, 11, 4077
  • Translated meta-GGA on-top functionals: PNAS 2024, 122, e2419413121
  • Global hybrid on-top functionals: JPCL 2020, 11, 10158 and JCTC 2020, 16, 2274
• Method extension publications:
  • XMS-PDFT: Faraday Discuss 2020, 224, 348-372
  • CMS-PDFT: JCTC 2020, 16, 7444
  • L-PDFT: JCTC 2023, 19, 3172
• Gradient publications:
  • CASSCF-PDFT: JCTC 2018, 14, 126
  • SA-CASSCF-PDFT: JCP 2020, 153, 014106 and JCP 2021, 154, 074108
  • CMS-PDFT: Mol. Phys. 2022, 120, e2110534
  • L-PDFT: JCTC 2024, 20, 3637
• Non-adiabatic coupling publications:
  • CMS-PDFT: JPCA 2024, 128, 1698

MC-PDFT has been hosted on PySCF-Forge for approximately three years as of the time of this writing. It includes a full battery of example and unit test scripts. The most recent major extension, the implementation of the MC23 functional of Bao et al. PNAS 2024, 122, e2419413121, is nearly six months old as of this writing.

Relevant Modules and Files

Transfer the following to PySCF with the same path:

• mcpdft/__init__.py
• mcpdft/mcpdft.py
• mcpdft/_libxc.py
• mcpdft/_dms.py
• mcpdft/otpd.py
• mcpdft/otfnal.py
• mcpdft/tfnal_derivs.py
• mcpdft/pdft_eff.py
• mcpdft/pdft_veff.py
• mcpdft/pdft_feff.py
• mcpdft/mspdft.py
• mcpdft/cmspdft.py
• mcpdft/xmspdft.py
• mcpdft/lpdft.py
• mcpdft/chkfile.py
• mcpdft/test/test_cmspdft.py
• mcpdft/test/test_pdft_veff.out
• mcpdft/test/test_diatomic_energies.py
• mcpdft/test/test_pdft_feff.py
• mcpdft/test/test_xmspdft.py
• mcpdft/test/test_pdft_veff.py
• mcpdft/test/test_spingaps_atoms.py
• mcpdft/test/test_mcpdft.py
• mcpdft/test/test_mgga.py
• mcpdft/test/test_otfnal.py
• mcpdft/test/test_otpd.py
• mcpdft/test/test_lpdft.py
• mcpdft/test/test_mspdft.py
• grad/mspdft.py
• grad/mcpdft.py
• grad/cmspdft.py
• grad/lpdft.py
• grad/test/h2co_tpbe66_631g_grad_num.npy
• grad/test/h2co_sa2_tpbe66_631g_grad_num.npy
• nac/mspdft.py
• df/grad/mspdft.py
• df/grad/mcpdft.py
• df/grad/lpdft.py

Transfer the following to PySCF with a possible change of path:

• grad/test/test_grad_mcpdft.py -> grad/test/test_mcpdft.py
• grad/test/test_grad_cmspdft.py -> grad/test/test_cmspdft.py
• grad/test/test_grad_h2co_slow.py -> grad/test/test_pdft_h2co_slow.py
• grad/test/test_grad_metagga_mcpdft.py -> grad/test/test_metagga_mcpdft.py
• grad/test/test_grad_h2co.py -> grad/test/test_pdft_h2co.py
• grad/test/test_grad_lpdft.py -> grad/test/test_lpdft.py
• grad/test/test_grad_mspdft.py -> grad/test/test_mspdft.py
• grad/test/test_diatomic_gradients.py -> grad/test/test_pdft_diatomic_gradients.py
• nac/test/test_nac_cmspdft.py -> nac/test/test_cmspdft.py
• mcpdft/nr_numint.c -> lib/mcpdft/nr_numint.c
• lib/dft/libxc_itrf2.c -> ??? (see below)
• dft2/libxc.py -> ??? (see below)

Note that lib/mcpdft/nr_numint.c will compile into libpdft.so and depends on libxc; the relevant part of PySCF-Forge's pyscf/lib/CMakeLists.txt must be transferred to PySCF.

Documentation

See pyscf.github.io #166

Long-term Maintenance Plan

Matt Hermes (@MatthewRHermes) can take the lead on long-term maintenance of MC-PDFT. Dayou Zhang (@Dayou-Zhang) can also provide support for maintaining the LibXC interface re-implementation (as necessary; see below).

Additional Context

The MC-PDFT implementation in PySCF-Forge notably includes a complete reimplementation of the core module's LibXC interface (pyscf/dft/libxc.py and pyscf/lib/dft/libxc_itrf.c), which was necessary in order to access LibXC's API for defining custom functionals as variants of existing functionals with modified parameters. This, in turn, was necessary because the original work on translated meta-GGA functionals PNAS 2024, 122, e2419413121 was inseparable from a functional optimization project which resulted in the custom on-top functional known as MC23.

The proper way to implement the functional-modification features of LibXC has been a subject of extended discussion. See:

• Implement interface to set custom parameters of XC functionals in Libxc pyscf#1746
• A new LibXC interface and new MC-DCFT code #70
  Notably, on August 27, 2023, @sunqm suggested the possibility that the implementation of such features might not be appropriate for PySCF at all.

Given the above, there are at least three possible ways to proceed:

1. Overwrite the PySCF LibXC interface with the MC-PDFT reimplementation.

• lib/dft/libxc_itrf2.c -> lib/dft/libxc_itrf.c
• dft2/libxc.py -> dft/libxc.py
• Pros:
  • Consolidate nearly-duplicate code.
  • Enable LibXC functional-modification features for all users of PySCF in all contexts.
• Cons:
  • Potentially disruptive and risky.
  • Requires experienced maintainers of PySCF core modules to re-learn the LibXC interface.

2. Move the duplicated interface ("dft2") into the PySCF core modules.

• lib/dft/libxc_itrf2.c -> lib/dft/libxc_itrf.c
• dft2/libxc.py -> dft2/libxc.py
• Or we could put them in the MC-PDFT folder.
• Pros:
  • Really easy.
  • Nondisruptive.
• Cons:
  • Bizarre near-duplication of code in PySCF will probably confuse new developers and give headaches.
  • Status quo labor-intensive manual synchronization of the two implementations still required.

3. Omit the LibXC interface reimplementation, and therefore the MC23 functional, from the transfer, and host these features only on PySCF-Forge for now.

• Pros:
  • Nondisruptive, at least to DFT.
  • Intuitively reasonable separation of concerns between the two projects.
• Cons:
  • Requires potentially error-prone partitioning of the existing MC-PDFT code into two parts, since the LibXC interface reimplementation is currently used in all MC-PDFT calculations.
  • Status quo manual resynchronization of the two implementations.

Other issues:

• MC-PDFT with XCFun #107: MC-PDFT currently has no support at all for the XCFun library.
• More efficient way for split_x_c #100: There are opportunities for improving computational efficiency.

[MatthewRHermes]

Personally, I advocate for option 1. The LibXC interface reimplementation was kept as close to the original implementation, seems to have similar performance to the original reimplementation, and enables new capacities for all users.

[sunqm]

+1 for option 1. If dcfnal overwrite commonly used functionals like pbe0, tpss, a warning might be needed.

[jeanwsr]

Option 1 looks good. And since pylibxc is not so recommended now, it seems the dft2 implementation can be one of the most practical choice. One more question: does the merge of option 1 break any functionality or API of pyscf-core's libxc interface?

[Dayou-Zhang]

So far the dft2 LibXC interface passes all LibXC unit tests in the core module. However, there are two edge cases that will break backward compatibility. I believe I can fix both places to make them fully backward compatible, but I am either uncertain whether we should support them, or I do not fully understand how to properly handle them. Any feedback is appreciated.

1. I dropped support for omega argument in eval_xc() and prompt users to use register_custom_functional_() if they wish to use a custom omega. I could support custom omega in this function by temporarily registering a custom functional and unregistering it after finishing the function call, but this would be inefficient for practical use, so it might be better to drop support and ask users to call register_custom_functional_() instead. In addition, omega would not work on custom functionals, since defining another custom functional on top of an existing custom functional is not supported.
2. I did not handle some edge cases in rsh_coeff() because I do not fully understand the reason behind the special handling of the edge cases that I commented out, and there is no test cases covering these edge cases. See the code in core module and dft2. I wonder if the contributor of this piece of code could provide additional explanation and perhaps add a few test cases.

In response to the comment by @sunqm, I will add a warning in register_custom_functional_() when users try to register a functional whose new_xc_code is already a valid built-in xc_code.

[jeanwsr]

For common usage of xc='RSH(...) + ...', the argument omega is not used. But for another way through mf.omega I guess it's used, as in this example

https://github.com/pyscf/pyscf/blob/07f1e2e8f9839964b13e4ee10919d014b42285c5/examples/dft/13-rsh_dft.py#L22

And there's a few tests for it, not in test_libxc but in test_h2o, test_gks, etc. Could dft2 pass those tests?

For the approach of implementation, is it possible to simply call _set_omega_density_threshold_ again in _eval_xc?

[MatthewRHermes]

pyscf-forge/pyscf/dft2/libxc.py

Lines 875 to 876 in 3dc2ed8

	def _eval_xc(xc_code, rho, spin=0, deriv=1, omega=None):
	xc = _get_xc(xc_code, spin)

Imagine inserting above line 876 something like

if omega is not None:
    xc_code = _edit_xc_code_omega (xc_code, omega)

which composes a new string related to the old string except that parse_xc on it resolves to a functional with the supplied omega. Then you get the best of both worlds: omega is changeable on eval_xc, but also all of Dayou's cacheing still works.

[Dayou-Zhang]

Looks like changeable omega in eval_xc is indeed an important feature. I will implement it.

I suppose calling _set_omega_density_threshold_() is probably the way to go. To ensure eval_xc still remains a pure function, I will cache the default omega value and call _set_omega_density_threshold_() again at the end of eval_xc to set omega back to its original value when users inputs a custom omega.

I will do a more thorough test over other unit tests in the core module to see if dft2 breaks any other modules.

[MatthewRHermes]

A context manager would be a good way to go about this as well

[MatthewRHermes]

For common usage of xc='RSH(...) + ...', the argument omega is not used. But for another way through mf.omega I guess it's used, as in this example

https://github.com/pyscf/pyscf/blob/07f1e2e8f9839964b13e4ee10919d014b42285c5/examples/dft/13-rsh_dft.py#L22

And there's a few tests for it, not in test_libxc but in test_h2o, test_gks, etc. Could dft2 pass those tests?

No.

(See this experimental branch corresponding to transfer option 1)

[Dayou-Zhang]

I just came up with a very elegant way to support omega in eval_xc(). Now the two compatibility issues are fixed. I have pushed the code in my personal branch, which is a fork of the mcpdft_1 branch by @MatthewRHermes . The code passes all unit tests.

@MatthewRHermes Shall I send a pull request to the mcpdft_1 branch in your account? I think it might be a good place to host the final draft of the code to be merged to the pyscf master branch. Do we also need to make changes to pyscf-forge?

[MatthewRHermes]

For the sake of organizational hygiene, I would suggest first advancing the PySCF-Forge implementation into a final form which includes all features which are to appear in the PySCF core branches, so that anyone (not only you or I) could perform the final transfer based only on the information in this thread.

Note also that there is to be a vote of the PySCF board on this proposal, if I understand correctly. I suppose other details of this transfer will also be discussed at that time.

[MatthewRHermes]

Fixed some issues with the list of test scripts in the proposal.