Together with Christoph Kreisbeck and Rafael A Molina I have contributed a blog entry to the News and Views section of the Journal of Physics describing our most recent work on AharonovBohm interferometer with an imbedded quantum dot (article, arxiv). Can you spot Schrödinger’s cat in the result?
Dusting off cometary surfaces: collimated jets despite a homogeneous emission pattern.
Knowledge of GPGPU techniques is helpful for rapid model building and testing of scientific ideas. For example, the beautiful pictures taken by the ESA/Rosetta spacecraft of comet 67P/Churyumov–Gerasimenko reveal jets of dust particles emitted from the comet. Wouldn’t it be nice to have a fast method to simulate thousands of dust particles around the comet and to find out if already the peculiar shape of this spacepotato influences the dusttrajectories by its gravitational potential? At the ZuseInstitut in Berlin we joined forces between the distributed algorithm and visual data analysis groups to test this idea. But first an accurate shape model of the comet 67P CG is required. As published in his blog, Mattias Malmer has done amazing work to extract a shapemodel from the published navigation camera images.
 Starting from the shape model by Mattias Malmer, we obtain a remeshed model with fewer triangles on the surface (we use about 20,000 triangles). The keyproperty of the new mesh is a homogeneous coverage of the cometary surface with almost equally sized triangle meshes. We don’t want better resolution and adaptive mesh sizes at areas with more complex features. Rather we are considering a homogeneous emission pattern without isolated activity regions. This is best modeled by mesh cells of equal area. Will this prescription yield nevertheless collimated dust jets? We’ll see…
 To compute the gravitational potential of such a surface we follow this nice article by JT Conway. The calculation later on stays in the rotating frame anchored to the comet, thus in addition the centrifugal and Coriolis forces need to be included.
 To accelerate the method, OpenCL comes to the rescue and lets one compute many trajectories in parallel. What is required are physical conditions for the starting positions of the dust as it flies off the surface. We put one dustparticle on the center of each triangle on the surface and set the initial velocity along the normal direction to typically 2 or 4 m/s. This ensures that most particles are able to escape and not fall back on the comet.
 To visualize the resulting point clouds of dust particles we have programmed an OpenGL visualization tool. We compute the rotation and sunlight direction on the comet to cast shadows and add activity profiles to the comet surface to mask out dust originating from the dark side of the comet.
This is what we get for May 3, 2015. The ESA/NAVCAM image is taken verbatim from the Rosetta/blog.
Read more about the physics and results in our arxiv article T. Kramer et al.: Homogeneous Dust Emission and Jet Structure near Active Cometary Nuclei: The Case of 67P/ChuryumovGerasimenko (submitted for publication) and grab the code to compute your own dust trajectories with OpenCL at github.org/noma/covis
Slow or fast transfer: bottleneck states in lightharvesting complexes
In the previous post I described some of the computational challenges for modeling energy transfer in the light harvesting complex II (LHCII) found in spinach. Here, I discuss the results we have obtained for the dynamics and choreography of excitonic energy transfer through the chlorophyll network. Compared to the FennaMatthewsOlson complex, LHCII has twice as many chlorophylls per monomeric unit (labeled 601614 with chlorophyll a and b types).
Previous studies of exciton dynamics had to stick to simple exponential decay models based on either Redfield or Förster theory for describing the transfer from the Chl b to the Chl a sites. The results are not satisfying and conclusive, since depending on the method chosen the transfer time differs widely (tens of picoseconds vs picoseconds!).
To resolve the discrepancies between the various approximate methods requires a more accurate approach. With the accelerated HEOM at hand, we revisited the problem and calculated the transfer rates. We find slower rates than given by the Redfield expressions. A combined FörsterRedfield description is possible in hindsight by using HEOM to identify a suitable cutoff parameter (Mcr=30/cm in this specific case).
Since the energy transfer is driven by the coupling of electronic degrees of freedom to vibrational ones, it of importance to assess how the vibrational mode distribution affects the transfer. In particular it has been proposed that specifically tuned vibrational modes might promote a fast relaxation. We find no strong impact of such modes on the transfer, rather we see (independent of the detailed vibrational structure) several bottleneck states, which act as a transient reservoir for the exciton flux. The details and distribution of the bottleneck states strongly depends on the parameters of the electronic couplings and differs for the two most commonly discussed LHCII models proposed by Novoderezhkin/Marin/van Grondelle and Müh/Madjet/Renger – both are considered in the article Scalable highperformance algorithm for the simulation of excitondynamics. Application to the light harvesting complex II in the presence of resonant vibrational modes (collaboration of Christoph Kreisbeck, Tobias Kramer, Alan AspuruGuzik).
Again, the correct assignment of the bottleneck states requires to use HEOM and to look beyond the approximate rate equations.
Highperformance OpenCL code for modeling energy transfer in spinach
With increasing computational power of massivelyparallel computers, a more accurate modeling of the energytransfer dynamics in larger and more complex photosynthetic systems (=lightharvesting complexes) becomes feasible – provided we choose the right algorithms and tools.
The diverse character of hardware found in highperformance computers (hpc) seemingly requires to rewrite program code from scratch depending if we are targeting multicore CPU systems, integrated manycore platforms (Xeon PHI/MIC), or graphics processing units (GPUs).
To avoid the defragmentation of our open quantumsystem dynamics workhorse (see the previous GPUHEOM posts) across the various hpcplatforms, we have transferred the GPUHEOM CUDA code to the Open Compute Language (OpenCL). The resulting QMaster tool is described in our just published article Scalable highperformance algorithm for the simulation of excitondynamics. Application to the light harvesting complex II in the presence of resonant vibrational modes (collaboration of Christoph Kreisbeck, Tobias Kramer, Alan AspuruGuzik). This post details the computational challenges and lessons learnt, the application to the lightharvesting complex II found in spinach will be the topic of the next post.
In my experience, it is not uncommon to develop a nice GPU application for instance with CUDA, which later on is scaled up to handle bigger problem sizes. With increasing problem size also the memory demands increase and even the 12 GB provided by the Kepler K40 are finally exhausted. Upon reaching this point, two options are possible: (a) to distribute the memory across different GPU devices or (b) to switch to architectures which provide more devicememory. Option (a) requires substantial changes to existing program code to manage the distributed memory access, while option (b) in combination with OpenCL requires (in the best case) only to adapt the kernellaunch configuration to the different platforms.
QMaster implements an extension of the hierarchical equation of motion (HEOM) method originally proposed by Tanimura and Kubo, which involves many (small) matrixmatrix multiplications. For GPU applications, the usage of local memory and the optimal threadgrids for fast matrixmatrix multiplications have been described before and are used in QMaster (and the publicly available GPUHEOM tool on nanohub.org). While for GPUs the best performance is achieved using shared/local memory and assign one thread to each matrix element, the multicore CPU OpenCL variant performs better with fewer threads, but getting more work per thread done. Therefore we use for the CPU machines a threadgrid which computes one complete matrix product per thread (this is somewhat similar to following the “naive” approach given in NVIDIA’s OpenCL programming guide, chapter 2.5). This strategy did not work very well for the Xeon PHI/MIC OpenCL case, which requires additional data structure changes, as we learnt from discussions with the distributed algorithms and hpc experts in the group of Prof. Reinefeld at the ZuseInstitute in Berlin.
The good performance and scaling across the 64 CPU AMD opteron workstation positively surprised us and lays the groundwork to investigate the validity of approximations to the energytransfer equations in the spinach lightharvesting system, the topic for the next post.
Tutorial #1: simulate 2d spectra of lightharvesting complexes with GPUHEOM @ nanoHub
The computation and prediction of twodimensional (2d) echo spectra of photosynthetic complexes is a daunting task and requires enormous computational resources – if done without drastic simplifications. However, such computations are absolutely required to test and validate our understanding of energy transfer in photosyntheses. You can find some background material in the recently published lecture notes on Modelling excitonicenergy transfer in lightharvesting complexes (arxiv version) of the Latin American School of Physics Marcos Moshinsky.
The ability to compute 2d spectra of photosynthetic complexes without resorting to strong approximations is to my knowledge an exclusive privilege of the Hierarchical Equations of Motion (HEOM) method due to its superior performance on massivelyparallel graphics processing units (GPUs). You can find some background material on the GPU performance in the two conference talks Christoph Kreisbeck and I presented at the GTC 2014 conference (recored talk, slides) and the first nanoHub users meeting.
 login on nanoHub.org (it’s free!)
 switch to the gpuheompop tool
 click the Launch Tool button (java required)
 for this tutorial we use the example input for “FMO coherence, 1 peak spectral density“.
You can select this preset from the Example selector.  we stick with the provided Exciton System parameters and only change the temperature to 77 K to compare the results with our published data.
 in the Spectral Density tab, leave all parameters at the the suggested values
 to compute 2d spectra, switch to the Calculation mode tab
 for compute: choose “twodimensional spectra“. This brings up inputmasks for setting the directions of all dipole vectors, we stick with the provided values. However, we select Rotational averaging: “four shot rotational average” and activate all six Liouville pathways by setting ground st[ate] bleach reph[asing , stim[ulated] emission reph[asing], and excited st[ate] abs[orption] to yes, as well as their nonrephasing counterparts (attention! this might require to resize the inputmask by pulling at the lower right corner)
 That’s all! Hit the Simulate button and your job will be executed on the carter GPU cluster at Purdue university. The simulation takes about 40 minutes of GPU time, which is orders of magnitude faster than any other published method with the same accuracy. You can close and reopen your session in between.
 Voila: your first FMO spectra appears.
 Now its time to change parameters. What happens at higher temperature?
 If you like the results or use them in your work for comparison, we (and the folks at nanoHub who generously develop and provide the nanoHub platform and GPU computation time) appreciate a citation. To make this step easy, a DOI number and reference information is listed at the bottom of the About tab of the toolpage.
With GPUHEOM we and now you (!) can not only calculate the 2d echo spectra of the FennaMatthewsOlson (FMO) complex, but also reveal the strong link between the continuum part of the vibrational spectral density and the prevalence of longlasting electronic coherences as written in my previous posts
GPU and cloud computing conferences in 2014
Two conferences are currently open for registration related to GPU and cloud computing. I will be attending and presenting at both, please email me if you want to get in touch at the meetings.
 April 911, 2014, talk about the GPUHEOM tool at the first nanoHUB users conference, Phoenix, Arizona
 March 2427, 2014, talk about GPU computing for understanding energytransfer in photosynthesis at the GPU Technology Conference 2014, San Jose, California
Oscillations in twodimensional spectroscopy
Over the last years, a debate is going on whether the observation of long lasting oscillatory signals in twodimensional spectra are reflecting vibrational of electronic coherences and how the functioning of the molecule is affected. Christoph Kreisbeck and I have performed a detailed theoretical analysis of oscillations in the FennaMatthewsOlson (FMO) complex and in a model threesite system. As explained in a previous post, the prerequisites for longlasting electronic coherences are two features of the continuous part of the vibronic mode density are: (i) a small slope towards zero frequency, and (ii) a coupling to the excitonic eigenenergy (ΔE) differences for relaxation. Both requirements are met by the mode density of the FMO complex and the computationally demanding calculation of twodimensional spectra of the FMO complex indeed predicts longlasting crosspeak oscillations with a period matching h/ΔE at room temperature (see our article LongLived Electronic Coherence in Dissipative ExcitonDynamics of LightHarvesting Complexes or arXiv version). The persistence of oscillations is stemming from a robust mechanism and does not require adding any additional vibrational modes at energies ΔE (the general background mode density is enough to support the relaxation toward a thermal state). But what happens if in addition to the background vibronic mode density additional vibronic modes are placed within the vicinity of the frequencies related electronic coherences? This finetuning model is sometimes discussed in the literature as an alternative mechanism for longlasting oscillations of vibronic nature. Again, the answer requires to actually compute twodimensional spectra and to carefully analyze the possible chain of lasermolecule interactions. Due to the special way twodimensional spectra are measured, the observed signal is a superposition of at least three pathways, which have different sensitivity for distinguishing electronic and vibronic coherences. Being a theoretical physicists now pays off since we have calculated and analyzed the three pathways separately (see our recent publication Disentangling Electronic and Vibronic Coherences in TwoDimensional Echo Spectra or arXiv version). One of the pathways leads to an enhancement of vibronic signals, while the combination of the remaining two diminishes electronic coherences otherwise clearly visible within each of them. Our conclusion is that estimates of decoherence times from twodimensional spectroscopy might actually underestimate the persistence of electronic coherences, which are helping the transport through the FMO network. The fine tuning and addition of specific vibrational modes leaves it marks at certain spots of the twodimensional spectra, but does not destroy the electronic coherence, which is still there as a Short Time Fourier Transform of the signal reveals.
Computational physics on GPUs: writing portable code
I am preparing my presentation for the simGPU meeting next week in Freudenstadt, Germany, and performed some benchmarks.
In the previous post I described how to get an OpenCL program running on a smartphone with GPU. By now Christoph Kreisbeck and I are getting ready to release our first smartphone GPU app for exciton dynamics in photosynthetic complexes, more about that in a future entry.
Getting the same OpenCL kernel running on laptop GPUs, workstation GPUs and CPUs, and smartphones/tablets is a bit tricky, due to different initialisation procedures and the differences in the optimal block sizes for the thread grid. In addition on a smartphone the local memory is even smaller than on a desktop GPU and doubleprecision floating point support is missing. The situation reminds me a bit of the “earlier days” of GPU programming in 2008.
Besides being a proof of concept, I see writing portable code as a sort of insurance with respect to further changes of hardware (however always with the goal to stick with the massively parallel programming paradigm). I am also amazed how fast smartphones are gaining computational power through GPUs!
Here some considerations and observations:

Standard CUDA code can be ported to OpenCL within a reasonable timeframe. I found the following resources helpful:
AMDs porting remarks
Matt Scarpinos OpenCL blog  The comparison of OpenCL vs CUDA performance for the same algorithm can reveal some surprises on NVIDIA GPUs. While on our C2050 GPU OpenCL works a bit faster for the same problem compared to the CUDA version, on a K20c system for certain problem sizes the OpenCL program can take several times longer than the CUDA code (no changes in the basic algorithm or workgroup sizes).
 The comparison with a CPU version running on 8 cores of the Intel Xeon machine is possible and shows clearly that the GPU code is always faster, but requires a certain minimal systems size to show its full performance.
 I am looking forward to running the same code on the Intel Xeon Phi systems now available with OpenCL drivers, see also this blog.
[Update June 22, 2013: I updated the graphs to show the 8core results using Intels latest OpenCL SDK. This brings the CPU runtimes down by a factor of 2! Meanwhile I am eagerly awaiting the possibility to run the same code on the Xeon Phis…]
Computational physics on the smartphone GPU
[Update August 2013: Google has removed the OpenCL library with Android 4.3. You can find an interesting discussion here. Google seems to push for its own renderscript protocol. I will not work with renderscript since my priorities are platform independency and sticking with widely adopted standards to avoid fragmentation of my code basis.]
I recently got hold of a Nexus 4 smartphone, which features a GPU (Qualcomm Adreno 320) and conveniently ships with already installed OpenCL library. With minimal changes I got the previously discussed manybody program code related to the fractional quantum Hall effect up and running. No unrooting of the phone is required to run the code example. Please use the following recipe at your own risk, I don’t accept any liabilities. Here is what I did:
 Download and unpack the Android SDK from google for crosscompilation (my host computer runs Mac OS X).
 Download and unpack the Android NDK from google to build minimal C/C++ programs without Java (no real app).

Install the standalone toolchain from the Android NDK. I used the following command for my installation:
/home/tkramer/androidndkr8d/build/tools/makestandalonetoolchain.sh \ installdir=/home/tkramer/androidndkstandalone
 Put the OpenCL programs and source code in an extra directory, as described in my previous post
 Change one line in the cl.hpp header: instead of including <GL/gl.h> change to <GLES/gl.h>. Note: I am using the “old” cl.hpp bindings 1.1, further changes might be required for the newer bindings, see for instance this helpful blog

Transfer the OpenCL library from the phone to a subdirectory lib/ inside your source code. To do so append the path to your SDK tools and use the adb command:
export PATH=/home/tkramer/adtbundlemacx86_6420130219/sdk/platformtools:$PATH adb pull /system/lib/libOpenCL.so

Cross compile your program. I used the following script, please feel free to provide shorter versions. Adjust the include directories and library directories for your installation.
rm plasma_disk_gpu /home/tkramer/androidndkstandalone/bin/armlinuxandroideabig++ v g \ DCL_USE_DEPRECATED_OPENCL_1_1_APIS DGPU \ I. \ I/home/tkramer/androidndkstandalone/include/c++/4.6 \ I/home/tkramer/androidndkr8d/platforms/android5/archarm/usr/include \ Llib \ march=armv7a mfloatabi=softfp mfpu=neon \ fpic fsignedchar fdatasections funwindtables fstackprotector \ ffunctionsections fdiagnosticsshowoption fPIC \ fnostrictaliasing fnoomitframepointer fnortti \ lOpenCL \ o plasma_disk_gpu plasma_disk.cpp

Copy the executable to the data dir of your phone to be able to run it. This can be done without rooting the phone with the nice SSHDroid App, which by defaults transfers to /data . Don’t forget to copy the kernel .cl files:
scp P 2222 integrate_eom_kernel.cl root@192.168.0.NNN: scp P 2222 plasma_disk_gpu root@192.168.0.NNN:
 ssh into your phone and run the GPU program:
ssh p 2222 root@192.168.0.NNN ./plasma_disk_gpu 64 16
 Check the resulting data files. You can copy them for example to the Download path of the storage and use the gnuplot (droidplot App) to plot them.
A short note about runtimes. On the Nexus 4 device the program runs for about 12 seconds, on a MacBook Pro with NVIDIA GT650M it completes in 2 seconds (in the example above the equations of motion for 16*64=1024 interacting particles are integrated). For larger particle numbers the phone often locks up.
An alternative way to transfer files to the device is to connect via USB cable and to install the Android Terminal Emulator app. Next
cd /data/data/jackpal.androidterm mkdir gpu chmod 777 gpu
On the host computer use adb to transfer the compiled program and the .cl kernel and start a shell to run the kernel
adb push integrate_eom_kernel.cl /data/data/jackpal.androidterm/gpu/ adb push plasma_disk_gpu /data/data/jackpal.androidterm/gpu/
You can either run the program within the terminal emulator or use the adb shell
adb shell cd /data/data/jackpal.androidterm/gpu/ ./plasma_disk_gpu 64 16
Let’s see in how many years todays desktop GPUs can be found in smartphones and which computational physics codes can be run!
Computational physics & GPU programming: exciton lab for lightharvesting complexes (GPUHEOM) goes live on nanohub.org
Christoph Kreisbeck and I are happy to announce the public availability of the Exciton Dynamics Lab for Light
Harvesting Complexes (GPUHEOM) hosted on nanohub.org. You need to register a user account (its free), and then you are ready to use GPUHEOM for the Frenkel exciton model of light harvesting complexes. In release 1.0 we support
 calculating population dynamics
 tracking coherences between two eigenstates
 obtaining absorption spectra
 twodimensional echo spectra (including excited state absorption)
 … and all this for general vibronic spectral densities parametrized by shifted Lorentzians.
I will post some more entries here describing how to use the tool for understanding how the spectral density affects the lifetime of electronic coherences (see also this blog entry).
In the supporting document section you find details of the implemented method and the assumptions underlying the tool. We are appreciating your feedback for further improving the tool.
We are grateful for the support of Prof. Gerhard Klimeck, Purdue University, director of the Network for Computational Nanotechnology to bring GPU computing to nanohub (I believe our tool is the first GPU enabled one at nanohub).
If you want to refer to the tool you can cite it as:
Christoph Kreisbeck; Tobias Kramer (2013), “Exciton Dynamics Lab for LightHarvesting Complexes (GPUHEOM),” https://nanohub.org/resources/gpuheompop. (DOI:10.4231/D3RB6W248).
and you find further references in the supporting documentation.
I very much encourage my colleagues developing computer programs for theoretical physics and chemistry to make them available on platforms such as nanohub.org. In my view, it greatly facilitates the comparison of different approaches and is the spirit of advancing science by sharing knowledge and providing reproducible data sets.
Good or bad vibrations for the FennaMatthewsOlson complex?
Due to its known structure and relative simplicity, the FennaMatthewsOlson complex of green sulfur bacteria provides an interesting testcase for our understanding of excitonic energy transfer in a lightharvesting complex.
The experimental pumpprobe spectra (discussed in my previous post catching and tracking light: following the excitations in the FennaMatthewsOlson complex) show longlasting oscillatory components and this finding has been a puzzle for theoretician and led to a refinement of the wellestablished models. These models show a reasonable agreement with the data and the rate equations explain the relaxation and transfer of excitonic energy to the reaction center.
However, the rate equations are based on estimates for the relaxation and dephasing rates. As Christoph Kreisbeck and I discuss in our article LongLived Electronic Coherence in Dissipative ExcitonDynamics of LightHarvesting Complexes (arxiv version), an exact calculation with GPUHEOM following the best data for the Hamiltonian allows one to determine where the simple approach is insufficient and to identify a keyfactor supporting electronic coherence:
It’s the vibronic spectral density – redrawn (in a different unit convention, multiplied by ω^{2}) from the article by M. Wendling from the group of Prof. Rienk van Grondelle. We did undertake a major effort to proceed in our calculations as close to the measured shape of the spectral density as the GPUHEOM method allows one. By comparison of results for different forms of the spectral density, we identify how the different parts of the spectral density lead to distinct signatures in the oscillatory coherences. This is illustrated in the figure on the rhs. To get long lasting oscillations and finally to relax, three ingredients are important
 a small slope towards zero frequency, which suppresses the pure dephasing.
 a high plateau in the region where the exciton energy differences are well coupled. This leads to relaxation.
 the peaked structures induce a “verylonglasting” oscillatory component, which is shown in the first figure. In our analysis we find that this is a persistent, but rather small (<0.01) modulation.
2d spectra are smart objects
The calculation of 2d echo spectra requires considerable computational resources. Since theoretically calculated 2d spectra are needed to check how well theory and experiment coincide, I conclude with showing a typical spectrum we obtain (including static disorder, but no excited state absorption for this example). One interesting finding is that 2d spectra are able to differentiate between the different spectral densities. For example for a a singlepeak DrudeLorentz spectral density (sometimes chosen for computational convenience), the wrong peaks oscillate and the lifetime of crosspeak oscillations is short (and becomes even shorter with longer vibronic memory). But this is for the experts only, see the supporting information of our article.
Are vibrations good or bad? Probably both… The pragmatic answer is that the FMO complex lives in an interesting parameter regime. The exact calculations within the Frenkel exciton model do confirm the wellknown dissipative energy transfer picture. But on the other hand the specific spectral density of the FMO complex supports longlived coherences (at least if the light source is a laser beam), which require considerable theoretical and experimental efforts to be described and measured. Whether the seen coherence has any biological relevance is an entirely different topic… maybe the greensulfur bacteria are just enjoying a glimpse into Schrödinger’s world of probabilistic uncertainty.
Computational physics & GPU programming: interacting manybody simulation with OpenCL
In the second example of my series on GPU programming for scientists, I discuss a short OpenCL program, which you can compile and run on the CPU and the GPUs of various vendors. This gives me the opportunity to perform some crossplatform benchmarks for a classical plasma simulation. You can expect dramatic (several 100 fold) speedups on GPUs for this type of system. This is one of the reasons why molecular dynamics code can gain quite a lot by incorporating the massively parallelprogramming paradigm in the algorithmic foundations.
The Open Computing Language (OpenCL) is relatively similar to its CUDA pendant, in practice the setup of an OpenCL kernel requires some housekeeping work, which might make the code look a bit more involved. I have based my interacting electrons calculation of transport in the Hall effect on an OpenCL code. Another examples is An OpenCL implementation for the solution of the timedependent Schrödinger equation on GPUs and CPUs (arxiv version) by C. Ó Broin and L.A.A. Nikolopoulos.
Now to the coding of a twodimensional plasma simulation, which is inspired by Laughlin’s mapping of a manybody wave function to an interacting classical ersatz dynamics (for some context see my short review Interacting electrons in a magnetic field: mapping quantum mechanics to a classical ersatzsystem on the arxiv).
Computational physics & GPU programming: Solving the timedependent Schrödinger equation
I start my series on the physics of GPU programming by a relatively simple example, which makes use of a mix of library calls and welldocumented GPU kernels. The runtime of the splitstep algorithm described here is about 280 seconds for the CPU version (Intel(R) Xeon(R) CPU E5420 @ 2.50GHz), vs. 10 seconds for the GPU version (NVIDIA(R) Tesla C1060 GPU), resulting in 28 fold speedup! On a C2070 the run time is less than 5 seconds, yielding an 80 fold speedup.
The description of coherent electron transport in quasi twodimensional electron gases requires to solve the Schrödinger equation in the presence of a potential landscape. As discussed in my post Time to find eigenvalues without diagonalization, our approach using wavepackets allows one to obtain the scattering matrix over a wide range of energies from a single wavepacket run without the need to diagonalize a matrix. In the following I discuss the basic example of propagating a wavepacket and obtaining the autocorrelation function, which in turn determines the spectrum. I programmed the GPU code in 2008 as a first test to evaluate the potential of GPGPU programming for my research. At that time doubleprecision floating support was lacking and the fast Fourier transform (FFT) implementations were little developed. Starting with CUDA 3.0, the program runs fine in double precision and my group used the algorithm for calculating electron flow through nanodevices. The CPU version was used for our articles in Physica Scripta Wave packet approach to transport in mesoscopic systems and the Physical Review B Phase shifts and phase πjumps in fourterminal waveguide AharonovBohm interferometers among others.
Here, I consider a very simple example, the propagation of a Gaussian wavepacket in a uniform potential V(x,y)=Fx, for which the autocorrelation function of the initial state
⟨x,yψ(t=0)⟩=1/(a√π)exp((x^{2}+y^{2})/(2 a^{2}))
is known in analytic form:
⟨ψ(t=0)ψ(t)⟩=2a^{2}m/(2a^{2}m+iℏt)exp(a^{2}F^{2}t^{2}/(4ℏ^{2})iF^{2}t^{3}/(24ℏ m)).
Continue reading “Computational physics & GPU programming: Solving the timedependent Schrödinger equation”
The physics of GPU programming
From discussions I learn that while many physicists have heard of Graphics Processing Units as fast computers, resistance to use them is widespread. One of the reasons is that physics has been relying on computers for a long time and tons of old, well trusted codes are lying around which are not easily ported to the GPU. Interestingly, the adoption of GPUs happens much faster in biology, medical imaging, and engineering.
I view GPU computing as a great opportunity to investigate new physics and my feeling is that todays methods optimized for serial processors may need to be replaced by a different set of standard methods which scale better with massively parallel processors. In 2008 I dived into GPU programming for a couple of reasons:
 As a “modelbuilder” the GPU allows me to reconsider previous limitations and simplifications of models and use the GPU power to solve the extended models.
 The turnaround time is incredibly fast. Compared to queues in conventional clusters where I wait for days or weeks, I get back results with 10000 CPU hours compute time the very same day. This in turn further facilitates the modelbuilding process.
 Some people complain about the strict synchronization requirements when running GPU codes. In my view this is an advantage, since essentially no messaging overhead exists.
 If you want to develop highperformance algorithm, it is not good enough to convert library calls to GPU library calls. You might get speedups of about 24. However, if you invest the time and develop your own knowhow you can expect much higher speedups of around 100 times or more, as seen in the applications I discussed in this blog before.
This summer I will lecture about GPU programming at several places and thus I plan to write a series of GPU related posts. I do have a complementary background in mathematical physics and special functions, which I find very useful in relation with GPU programming since new physical models require a stringent mathematical foundation and numerical studies.
Catching and tracking light: following the excitations in the FennaMatthewsOlson complex
Efficient and fast transport of electric current is a basic requirement for the functioning of nanodevices and biological systems. A neat example is the energytransport of a lightinduced excitation in the FennaMatthewsOlson complex of green sulfur bacteria. This process has been elucidated by pumpprobe spectroscopy. The resulting spectra contain an enormous amount of information about the couplings of the different pigments and the pathways taken by the excitation. The basic guide to a 2d echospectrum is as follows:
You can find peaks of high intensity along the diagonal line which are roughly representing a more common absorption spectrum. If you delay the pump and probe pulses by several picoseconds, you will find a new set of peaks at a horizontal axis which indicates that energy of the excitation gets redistributed and the system relaxes and transfers part of the energy to vibrational motion. This process is nicely visible in the spectra recorded by Brixner et al.
A lot of excitement and activity on photosynthetic complexes was triggered by experiments of Engel et al showing that besides the relaxation process also periodic oscillations are visible in the oscillations for more than a picosecond.
What is causing the oscillations in the peak amplitudes of 2d echospectra in the FennaMatthews Olson complex?
A purely classical transport picture should not show such oscillations and the excitation instead hops around the complex without interference. Could the observed oscillations point to a different transport mechanism, possibly related to the quantummechanical simultaneous superposition of several transport paths?
The initial answer from the theoretical side was no, since within simplified models the thermalization occurs fast and without oscillations. It turned out that the simple calculations are a bit too simplistic to describe the system accurately and exact solutions are required. But exact solutions (even for simple models) are difficult to obtain. Known exact methods such as DMRG work only reliable at very low temperatures (273 C), which are not directly applicable to biological systems. Other schemes use the famous path integrals but are too slow to calculate the pumpprobe signals.
Our contribution to the field is to provide an exact computation of the 2d echospectra at the relevant temperatures and to see the difference to the simpler models in order to quantify how much coherence is preserved. From the methoddevelopment the computational challenge is to speedup the calculations several hundred times in order to get results within days of computational runtime. We did achieve this by developing a method which we call GPUhierarchical equations of motion (GPUHEOM). The hierarchical equations of motions are a nice scheme to propagate a density matrix under consideration of nonMarkovian effects and strong couplings to the environment. The HEOM scheme was developed by Kubo, Tanimura, and Ishizaki (Prof. Tanimura has posted some material on HEOM here).
However, the original computational method suffers from the same problems as pathintegral calculations and is rather slow (though the HEOM method can be made faster and applied to electronic systems by using smart filtering as done by Prof. YiJing Yan). The GPU part in GPUHEOM stands for Graphics Processing Units. Using our GPU adoption of the hierarchical equations (see details in Kreisbeck et al[JCTC, 7, 2166 (2011)] ) allowed us to cut down computational times dramatically and made it possible to perform a systematic study of the oscillations and the influence of temperature and disorder in our recent article Hein et al [New J. of Phys., 14, 023018 (2012), open access] .
Time to find eigenvalues without diagonalization
Solving the stationary Schrödinger (HE)Ψ=0 equation can in principle be reduced to solving a matrix equation. This eigenvalue problem requires to calculate matrix elements of the Hamiltonian with respect to a set of basis functions and to diagonalize the resulting matrix. In practice this time consuming diagonalization step is replaced by a recursive method, which yields the eigenfunctions for a specific eigenvalue.
A very different approach is followed by wavepacket methods. It is possible to propagate a wavepacket without determining the eigenfunctions beforehand. For a given Hamiltonian, we solve the timedependent Schrödinger equation (i ∂tH) Ψ=0 for an almost arbitrary initial state Ψ(t=0) (initial value problem).
The reformulation of the determination of eigenstates as an initial value problem has a couple of computational advantages:
 results can be obtained for the whole range of energies represented by the wavepacket, whereas a recursive scheme yields only one eigenenergy
 the wavepacket motion yields direct insight into the pathways and allows us to develop an intuitive understanding of the transport choreography of a quantum system
 solving the timedependent Schrödinger equation can be efficiently implemented using Graphics Processing Units (GPU), resulting in a large (> 20 fold) speedup compared to CPU code
The determination of transmissions requires now to calculate the Fourier transform of correlation functions <Ψ(t=0)Ψ(t)>. This method has been pioneered by Prof. Eric J. Heller, Harvard University, and I have written an introductory article for the Latin American School of Physics 2010 (arxiv version).
Recently, Christoph Kreisbeck has done a detailed calculations on the gatevoltage dependency of the conductance in AharonovBohm nanodevices, taking full adventage of the simultaneous probing of a range of Fermi energies with one single wavepacket. A very clean experimental realization of the device was achieved by Sven Buchholz, Prof. Saskia Fischer, and Prof. Ulrich Kunze (RU Bochum), based on a semiconductor material grown by Dr. Dirk Reuter and Prof. Anreas Wieck (RU Bochum). The details, including a comparison of experimental and theoretical results shown in the left figure, are published in Physical Review B (arxiv version).
Interactions: from galaxies to the nanoscale
For a while we have explored the usage of General Purpose Graphics Processing Units (GPGPU) for electronic transport calculations in nanodevices, where we want to include all electronelectron and electrondonor interactions. The GPU allows us to drastically (250 fold !!!) boost the performance of Nbody codes and we manage to propagate 10,000 particles over several million timesteps within days. While GPU methods are now rather popular within the astrophysics crowd, we haven’t seen many GPU applications for electronic transport in a nanodevice. Besides the change from astronomical units to atomic ones, gravitational forces are always attractive, whereas electrons are affected by electrondonor charges (attractive) and electronelectron repulsion. Furthermore we have a magnetic field present, leading to deflections. Last, the space where electrons can spread out is limited by the device borders. In total the force on the kth electron is given by
Our recent paper in Physical Review B (also freely available on the arxiv) gives the first microscopic description of the classical Hall effect, where interactions are everything: without interactions no Hall field and no drift transport. The role and importance of the interactions is surprisingly sparsely mentioned in the literature, probably due to a lack of computational means to move beyond phenomenological models. A notable exception is the very first paper on the Hall effect by Edwin Hall, where he writes “the phenomena observed indicate that two currents, parallel and in the same direction, tend to repel each other”. Note that this repulsion works throughout the device and therefore electrons do not pile up at the upper edge, but rather a complete redistribution of the electronic density takes place, yielding the potential shown in the figure.
Another important part of our simulation of the classical Hall effect are the electron sources and sinks, the contacts at the left and right ends of the device. We have developed a feedin and removal model of the contacts, which keeps the contact on the same (externally enforced) potential during the course of the simulation.
Mindboggling is the fact that the very same “classical Hall potential” has also been observed in conjunction with a plateau of the integer quantum Hall effect (IQHE) [Knott et al 1995 Semicond. Sci. Technol. 10 117 (1995)]. Despite these observations, many theoretical models of the integer quantum Hall effect do not consider the interactions between the electrons. In our classical model, the Hall potential for noninteracting electrons differs dramatically from the solution shown above and transport proceeds then (and only then) along the lower and upper edges. However the edge current solution is not compatible with the contact potential model described above where an external reservoir enforces equipotentials within each contact.