Debugging Nextfow core/plugins

Say you’re developing your own Nextflow plugin (or want to learn how Nextflow works internally) and don’t want to populate your code with thousands of println "passing 1".

Say you want to debug your code to understand these corner cases that tests are not able to reproduce

In this post, I’ll show you how easily it is using Intellij (Community Edition is ok). So basically the requirements are:

I’ll use nf-parquet as the plugin I want to debug

In a fresh directory you need to grab the project

git clone https://github.com/nextflow-io/nf-parquet

Next, we need to build and install the plugin in our machine

./gradlew installPlugin

This command will compile and install last version of the plugin in our $HOME/.nextflow/plugins folder

Open the nf-parquet directory with Intellij and let it to reindex the project (hope a few seconds)

In the Project view (at the left) navigate and open ParquetExtension.groovy file

Scroll down until line 44 and click on the line number. A red dot will show the breakpoint is activated in this line

Now we need to create a "JVM Remote Debug configuration".

Select "Edit configuration":

and create a new JVM Debug configuration:

press OK and accept default values

nf-parquet comes with some simple pipelines to validate it. Open a console terminal and navigate to the validation folder and execute one of them:

If all goes well, the pipeline will be stopped at the start and is waiting for the debug session showing the message Listening for transport dt_socket at address: 5005

Go to the Intellij editor and click in the "bug" green button close to the "Unnamed" configuration (in case you didnt provided one)

As soon both JVMs are connected your pipeline will start. When the execution reach the breakpoint it will be stopped and you can inspect the variables, for example:

Uses the debug buttons to "continue", "step by step", etc to continue with the execution of your pipeline

Say you find where your code is failing, or you want to try something new. Simple change your code, execute again the installPlugin task and repeat the process